2024-03-25T17:51:17.252 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:51:17.253 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/orig.config.yaml 2024-03-25T17:51:17.295 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:51:17.364 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621309' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUEFSADnVvILqzvtrWtHAF9jYxUwPIXtnJtROHs2BTLVMNONytR0y9dNqmYCAahSYtKDAkrpvk18RnY3FK4g8Q= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBq7nuCIIYPLhS6F1kW7ValjeH/QJoJLAYq3KIIS6zBJhjXFs5rWzRdx5QRsovxS0X92BplHJpKsJ1Ln2Eo+s/M= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:51:17.364 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:51:17.365 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:51:17.365 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:51:17.367 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:51:17.390 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:51:17.390 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:51:17.390 INFO:teuthology.packaging:ref: None 2024-03-25T17:51:17.390 INFO:teuthology.packaging:tag: None 2024-03-25T17:51:17.390 INFO:teuthology.packaging:branch: main 2024-03-25T17:51:17.390 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:51:17.390 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:51:17.749 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:51:17.750 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:51:17.759 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:51:17.759 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:51:17.769 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:51:17.784 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:51:17.791 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:51:17.814 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:11.561303', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUEFSADnVvILqzvtrWtHAF9jYxUwPIXtnJtROHs2BTLVMNONytR0y9dNqmYCAahSYtKDAkrpvk18RnY3FK4g8Q='} 2024-03-25T17:51:17.836 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:11.562925', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBq7nuCIIYPLhS6F1kW7ValjeH/QJoJLAYq3KIIS6zBJhjXFs5rWzRdx5QRsovxS0X92BplHJpKsJ1Ln2Eo+s/M='} 2024-03-25T17:51:17.836 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:51:17.844 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:51:17.844 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:51:17.844 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:51:17.912 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe221a110d0>, signals=[15]) 2024-03-25T17:51:17.912 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:51:17.923 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:51:17.923 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T17:51:17.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:18.015 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T17:51:18.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:51:18.104 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:51:18.112 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-03-25T17:51:18.129 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-03-25T17:51:18.129 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-03-25T17:51:18.184 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream" 2024-03-25T17:51:18.184 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:18.185 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:18.186 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-03-25T17:51:18.206 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-03-25T17:51:18.223 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-03-25T17:51:18.223 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:51:18.280 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:51:18.281 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:51:18.281 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-03-25T17:51:18.305 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:51:18.314 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:51:18.351 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:51:18.351 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:18.354 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:51:18.369 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:51:18.375 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:51:18.375 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:18.410 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:51:18.518 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:51:18.563 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:51:18.580 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:51:18.580 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:51:18.581 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:18.581 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:51:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:18.597 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-03-25T17:51:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:51:18.598 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-25T17:51:18.654 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:18.654 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.14.0-361.el9.x86_64 2024-03-25T17:51:18.654 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum install -y kernel 2024-03-25T17:51:18.655 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:51:18.655 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-361.el9.x86_64 2024-03-25T17:51:18.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-03-25T17:51:21.347 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 6.9 MB/s | 6.7 MB 00:00 2024-03-25T17:51:21.588 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - AppStream 5.1 MB/s | 6.7 MB 00:01 2024-03-25T17:51:22.789 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-25T17:51:23.346 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-25T17:51:31.396 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 994 kB/s | 8.0 MB 00:08 2024-03-25T17:51:31.569 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.0 MB 00:07 2024-03-25T17:51:40.654 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-25T17:51:40.892 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 19 MB 00:08 2024-03-25T17:51:51.864 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 827 kB/s | 6.1 MB 00:07 2024-03-25T17:51:53.213 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 718 kB/s | 6.1 MB 00:08 2024-03-25T17:51:53.669 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-25T17:51:54.841 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-25T17:51:55.531 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T17:52:00.471 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T17:52:02.602 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:52:02.603 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:52:02.704 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T17:52:02.716 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-03-25T17:52:02.717 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:52:02.717 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 76 M 2024-03-25T17:52:02.717 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 123 M 2024-03-25T17:52:02.717 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T17:52:02.871 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:02.873 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:02.873 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:02.986 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:02.996 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:03.005 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:03.021 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:03.030 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:03.039 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:03.214 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:03.224 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:03.231 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:03.650 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:03.675 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:03.702 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:03.789 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:03.818 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:03.830 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:03.964 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:03.975 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:03.995 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:04.113 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:04.127 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:04.150 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:04.191 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:04.209 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:04.216 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:04.243 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:04.252 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:04.284 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:04.285 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:04.318 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:04.337 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:04.753 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:04.781 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:04.806 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:05.081 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:05.101 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:05.125 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:05.126 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.159 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.233 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.303 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.316 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.390 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:05.541 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.541 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.560 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.561 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.579 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.598 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:05.625 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.626 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.706 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.768 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.793 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.902 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:05.912 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:05.929 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:05.999 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:06.062 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:06.074 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:06.083 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:06.118 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:06.156 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:06.171 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:06.177 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:06.228 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:06.237 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:06.278 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:06.278 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:06.280 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:06.319 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:06.328 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:06.330 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:06.370 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:06.370 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:06.411 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:06.430 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:06.430 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:06.489 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:06.580 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 18 kB/s | 66 kB 00:03 2024-03-25T17:52:06.672 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:06.714 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:07.022 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.6 MB/s | 15 MB 00:04 2024-03-25T17:52:07.025 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:07.062 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:07.151 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:07.581 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.0 MB/s | 33 MB 00:04 2024-03-25T17:52:07.904 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:08.062 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:08.252 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:08.269 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:08.398 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:08.926 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:08.980 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:09.026 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:09.391 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:09.586 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:09.681 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:09.746 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:09.765 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:09.840 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:09.965 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:09.985 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:10.127 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:10.184 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:10.256 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:10.303 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:52:13.038 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 4.3 MB/s | 27 MB 00:06 2024-03-25T17:52:13.045 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:13.045 INFO:teuthology.orchestra.run.smithi203.stdout:Total 7.4 MB/s | 76 MB 00:10 2024-03-25T17:52:13.710 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T17:52:14.116 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T17:52:14.116 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T17:52:15.118 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T17:52:15.118 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T17:52:17.586 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T17:52:18.200 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:18.283 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:19.068 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 879 kB/s | 21 MB 00:24 2024-03-25T17:52:21.026 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:21.171 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:23.898 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-25T17:52:25.868 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:25.913 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:26.063 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:52:26.064 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:52:26.157 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:52:26.167 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:52:26.168 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T17:52:26.168 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T17:52:26.168 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T17:52:26.168 INFO:teuthology.orchestra.run.smithi143.stdout:Install 4 Packages 2024-03-25T17:52:26.168 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T17:52:26.168 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 76 M 2024-03-25T17:52:26.168 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 123 M 2024-03-25T17:52:26.169 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T17:52:26.330 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:26.331 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:26.332 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:26.453 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:26.461 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:26.475 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:26.494 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:26.501 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:26.515 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:26.621 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:26.626 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:26.646 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:26.687 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:26.693 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:26.716 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:26.828 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:26.837 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:26.846 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:26.854 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:26.903 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:26.912 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:26.922 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:26.922 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:26.974 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:27.072 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:27.083 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:27.134 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:27.421 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:27.443 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:27.480 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:27.486 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:27.509 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:27.568 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:27.626 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:27.632 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:27.699 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:27.708 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:27.714 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:27.717 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:27.747 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:27.766 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:27.783 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:27.818 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:27.838 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:27.916 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:27.994 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:28.132 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:28.256 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:28.279 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:28.754 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:28.764 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:28.764 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:28.784 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:28.785 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:28.890 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:28.892 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:28.928 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:28.965 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:28.967 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:29.002 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:29.004 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:29.020 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:29.051 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:29.089 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:29.108 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:29.331 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:29.476 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:29.602 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:29.742 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:30.017 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:30.132 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:30.179 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:30.424 INFO:teuthology.orchestra.run.smithi143.stdout:(1/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.6 MB/s | 15 MB 00:04 2024-03-25T17:52:30.426 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:30.475 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:30.538 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:30.562 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:30.592 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:52:30.679 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:30.719 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:52:30.750 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:30.757 INFO:teuthology.orchestra.run.smithi143.stdout:(2/4): kernel-5.14.0-432.el9.x86_64.rpm 15 kB/s | 66 kB 00:04 2024-03-25T17:52:30.861 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:30.983 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:52:31.061 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:31.391 INFO:teuthology.orchestra.run.smithi143.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.6 MB/s | 33 MB 00:05 2024-03-25T17:52:31.399 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:31.809 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:31.975 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:52:32.098 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:32.164 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:52:32.181 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:32.237 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:52:32.258 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:32.293 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:52:32.368 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:32.441 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:52:32.562 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:32.701 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:52:33.185 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:52:33.203 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:52:33.365 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:52:33.958 INFO:teuthology.orchestra.run.smithi143.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 7.8 MB/s | 27 MB 00:03 2024-03-25T17:52:33.963 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:33.963 INFO:teuthology.orchestra.run.smithi143.stdout:Total 9.7 MB/s | 76 MB 00:07 2024-03-25T17:52:34.629 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T17:52:34.997 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T17:52:34.997 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T17:52:36.019 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T17:52:36.019 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T17:52:38.476 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T17:52:39.286 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:52:39.370 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:42.112 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:52:42.249 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:46.931 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:52:47.058 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:51.835 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:59.770 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:52:59.770 INFO:teuthology.orchestra.run.smithi203.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:52:59.770 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:52:59.815 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:53:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:53:00.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:53:00.618 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:00.618 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:53:00.618 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T17:53:00.618 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:00.618 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:53:00.618 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:53:00.619 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:53:00.619 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:53:00.619 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T17:53:00.979 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:01.074 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:01.075 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:01.075 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:53:01.076 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:53:01.076 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:53:01.076 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:53:01.076 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:53:01.076 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:01.156 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:01.156 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:01.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-03-25T17:53:01.190 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 68.el9 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8264254 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-03-25T17:53:01.191 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T17:53:01.192 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:53:01.193 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:53:01.193 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:53:01.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:53:01.830 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-03-25T17:53:03.223 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:53:03.241 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-03-25T17:53:03.244 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-25T17:53:03.259 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.EiZd8Is5nM 2024-03-25T17:53:03.259 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EiZd8Is5nM 2024-03-25T17:53:03.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.EiZd8Is5nM 2024-03-25T17:53:03.473 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.EiZd8Is5nM is 6KB 2024-03-25T17:53:03.523 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.EiZd8Is5nM 2024-03-25T17:53:03.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:53:03.602 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:53:03.602 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:53:03.602 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:53:03.602 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:53:03.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:53:03.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-03-25T17:53:21.189 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:21.189 INFO:teuthology.orchestra.run.smithi143.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:53:21.189 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T17:53:21.233 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:21.583 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:21.583 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:53:21.583 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:53:21.583 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:53:22.020 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:53:22.021 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T17:53:22.021 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T17:53:22.021 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:22.021 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:53:22.021 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:53:22.021 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:53:22.021 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T17:53:22.021 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T17:53:22.381 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:22.478 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:22.478 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:22.478 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:53:22.478 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:53:22.478 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:53:22.479 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:53:22.479 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:53:22.479 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:53:22.558 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:53:22.558 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:53:22.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo rpm -qi grub2-tools 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:Name : grub2-tools 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:Epoch : 1 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:Version : 2.06 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:Release : 68.el9 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: x86_64 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:Group : Unspecified 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:Size : 8264254 2024-03-25T17:53:22.601 INFO:teuthology.orchestra.run.smithi143.stdout:License : GPLv3+ 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:Packager : builder@centos.org 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:Vendor : CentOS 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:Summary : Support tools for GRUB. 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:Description : 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:hardware devices. 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T17:53:22.602 INFO:teuthology.orchestra.run.smithi143.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:53:22.603 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:53:22.603 INFO:teuthology.task.kernel:Updating grub on smithi143 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:53:22.603 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:53:23.188 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-03-25T17:53:24.565 INFO:teuthology.orchestra.run.smithi143.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:53:24.597 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-03-25T17:53:24.600 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-03-25T17:53:24.614 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.xRtApIN58l 2024-03-25T17:53:24.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xRtApIN58l 2024-03-25T17:53:24.688 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.xRtApIN58l 2024-03-25T17:53:24.829 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.xRtApIN58l is 6KB 2024-03-25T17:53:24.851 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.xRtApIN58l 2024-03-25T17:53:24.865 DEBUG:teuthology.orchestra.run.smithi143:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:53:24.930 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:53:24.930 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:53:24.930 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:53:24.930 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:53:24.931 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:53:25.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-03-25T17:53:33.758 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:53:33.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T17:53:33.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:53:55.097 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:53:55.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-25T17:53:55.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:08.287 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-25T17:54:17.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T17:54:17.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:35.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-25T17:54:47.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T17:54:47.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:54:50.804 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-25T17:54:51.282 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T17:54:51.283 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:54:51.283 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-25T17:54:51.297 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:54:51.297 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:54:51.297 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:54:51.297 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:54:52.298 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:54:52.298 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:54:52.393 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-03-25T17:54:52.417 DEBUG:teuthology.parallel:result is None 2024-03-25T17:54:55.100 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:55:04.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-25T17:55:04.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:55:04.760 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-25T17:55:05.230 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-25T17:55:05.230 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:55:05.230 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-03-25T17:55:05.244 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:55:05.244 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:55:05.244 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:55:05.244 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:55:06.245 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:55:06.245 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:55:06.307 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-03-25T17:55:06.339 DEBUG:teuthology.parallel:result is None 2024-03-25T17:55:06.340 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:55:06.347 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:55:06.347 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:55:06.350 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:55:06.370 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:55:06.378 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:55:06.388 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:55:06.388 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:55:06.405 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:55:06.499 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:55:06.508 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:55:06.508 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:55:06.510 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:55:06.543 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:06.544 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:06.556 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:06.558 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:55:06.560 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:55:06.568 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:55:06.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:55:06.600 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:55:06.630 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:55:06.639 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:55:06.639 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:55:06.668 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:55:06.685 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:06.765 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:06.810 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:06.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:06.933 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T17:55:06.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:55:07.002 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:07.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:55:07.100 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:07.181 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:55:07.226 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:55:07.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:55:07.295 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-03-25T17:55:07.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-03-25T17:55:07.346 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:55:07.393 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:55:07.763 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:55:07.772 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:55:07.773 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:55:07.784 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:55:07.794 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2024-03-25T17:55:07.881 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2024-03-25T17:55:07.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-03-25T17:55:07.948 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-03-25T17:55:07.950 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:55:07.950 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2024-03-25T17:55:07.976 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2024-03-25T17:55:07.976 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-03-25T17:55:07.997 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-03-25T17:55:07.997 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi143.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:55:07.998 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:55:08.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:55:08.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:55:08.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:55:08.070 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:55:08.070 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:55:08.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:55:08.140 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:55:08.150 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T17:55:08.156 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T17:55:08.281 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:55:08.298 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:55:08.301 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi143.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T17:59:44.719 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi143.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-03-25T17:59:44.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-25T17:59:44.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:59:44.853 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-25T17:59:44.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-25T17:59:44.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T17:59:44.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:59:45.027 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-25T17:59:45.106 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-25T17:59:45.106 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T17:59:45.119 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T17:59:45.119 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:59:45.120 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:59:45.122 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:59:45.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:59:45.152 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:59:45.169 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:59:45.177 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:59:45.194 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:59:45.200 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2024-03-25T17:59:45.214 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2024-03-25T17:59:45.223 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-03-25T17:59:45.230 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:59:45.237 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-03-25T17:59:45.246 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:59:45.254 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:59:45.270 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:59:45.303 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:59:45.307 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:59:45.307 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-03-25T17:59:45.307 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:45.307 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:45.307 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:45.308 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:45.326 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:59:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:59:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-25T17:59:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:59:45.331 INFO:teuthology.run_tasks:Running task install... 2024-03-25T17:59:45.340 DEBUG:teuthology.task.install:project ceph 2024-03-25T17:59:45.340 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T17:59:45.340 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T17:59:45.340 INFO:teuthology.task.install:Using flavor: default 2024-03-25T17:59:45.346 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T17:59:45.346 INFO:teuthology.task.install:extra packages: [] 2024-03-25T17:59:45.350 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:59:45.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:59:45.353 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:59:45.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:59:45.497 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:59:45.498 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:59:45.593 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:59:45.593 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:59:45.615 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:59:45.616 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T17:59:45.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:59:45.644 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:59:45.645 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:59:45.712 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:59:45.736 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:59:45.736 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T17:59:45.737 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:59:45.765 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:59:45.765 DEBUG:teuthology.orchestra.run.smithi143:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:59:45.791 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:59:45.832 DEBUG:teuthology.orchestra.run.smithi143:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:59:45.859 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-03-25T17:59:45.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-03-25T17:59:45.912 DEBUG:teuthology.orchestra.run.smithi143:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:59:45.978 INFO:teuthology.orchestra.run.smithi143.stdout:check_obsoletes = 1 2024-03-25T17:59:45.980 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum clean all 2024-03-25T17:59:46.111 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-03-25T17:59:46.139 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-03-25T17:59:46.211 INFO:teuthology.orchestra.run.smithi143.stdout:74 files removed 2024-03-25T17:59:46.238 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw 2024-03-25T17:59:47.278 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 98 kB/s | 82 kB 00:00 2024-03-25T17:59:47.639 INFO:teuthology.orchestra.run.smithi143.stdout:ceph packages for x86_64 74 kB/s | 82 kB 00:01 2024-03-25T17:59:48.138 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T17:59:48.739 INFO:teuthology.orchestra.run.smithi143.stdout:ceph noarch packages 9.5 kB/s | 10 kB 00:01 2024-03-25T17:59:49.029 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T17:59:49.448 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T17:59:49.798 INFO:teuthology.orchestra.run.smithi143.stdout:ceph source packages 1.9 kB/s | 1.9 kB 00:01 2024-03-25T17:59:50.217 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T17:59:50.856 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-25T17:59:51.759 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-25T17:59:59.338 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 1.0 MB/s | 8.0 MB 00:08 2024-03-25T18:00:00.340 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 999 kB/s | 8.0 MB 00:08 2024-03-25T18:00:09.083 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 19 MB 00:08 2024-03-25T18:00:11.084 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-03-25T18:00:21.521 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 706 kB/s | 6.1 MB 00:08 2024-03-25T18:00:22.580 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 795 kB/s | 6.1 MB 00:07 2024-03-25T18:00:23.476 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-25T18:00:24.009 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-03-25T18:00:24.646 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-25T18:00:25.186 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-25T18:00:29.593 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T18:00:30.169 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T18:00:32.051 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:00:32.058 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:00:32.058 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:00:32.058 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:00:32.058 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:00:32.058 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:00:32.058 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:00:32.059 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:00:32.060 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:00:32.061 INFO:teuthology.orchestra.run.smithi203.stdout:Install 30 Packages 2024-03-25T18:00:32.062 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-03-25T18:00:32.062 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:00:32.062 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 58 M 2024-03-25T18:00:32.062 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:00:32.591 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:00:32.604 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:00:32.604 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:00:32.604 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:00:32.604 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:00:32.604 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:00:32.604 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrading: 2024-03-25T18:00:32.604 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:00:32.605 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:00:32.606 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout:Installing weak dependencies: 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout:Install 30 Packages 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade 2 Packages 2024-03-25T18:00:32.607 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:00:32.608 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 58 M 2024-03-25T18:00:32.608 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:00:41.761 INFO:teuthology.orchestra.run.smithi143.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 611 kB/s | 5.3 MB 00:08 2024-03-25T18:00:41.861 INFO:teuthology.orchestra.run.smithi143.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T18:00:43.361 INFO:teuthology.orchestra.run.smithi143.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 495 kB/s | 742 kB 00:01 2024-03-25T18:00:44.836 INFO:teuthology.orchestra.run.smithi143.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 338 kB/s | 498 kB 00:01 2024-03-25T18:00:53.068 INFO:teuthology.orchestra.run.smithi203.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 265 kB/s | 5.3 MB 00:20 2024-03-25T18:00:53.984 INFO:teuthology.orchestra.run.smithi203.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 27 kB/s | 25 kB 00:00 2024-03-25T18:00:55.685 INFO:teuthology.orchestra.run.smithi143.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 481 kB/s | 5.1 MB 00:10 2024-03-25T18:00:55.868 INFO:teuthology.orchestra.run.smithi143.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 246 kB/s | 45 kB 00:00 2024-03-25T18:00:56.194 INFO:teuthology.orchestra.run.smithi143.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 404 kB/s | 131 kB 00:00 2024-03-25T18:00:56.610 INFO:teuthology.orchestra.run.smithi143.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 389 kB/s | 162 kB 00:00 2024-03-25T18:00:57.193 INFO:teuthology.orchestra.run.smithi203.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 231 kB/s | 742 kB 00:03 2024-03-25T18:00:57.269 INFO:teuthology.orchestra.run.smithi143.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 487 kB/s | 320 kB 00:00 2024-03-25T18:00:57.994 INFO:teuthology.orchestra.run.smithi143.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 411 kB/s | 298 kB 00:00 2024-03-25T18:00:58.336 INFO:teuthology.orchestra.run.smithi143.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 291 kB/s | 99 kB 00:00 2024-03-25T18:00:58.445 INFO:teuthology.orchestra.run.smithi143.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-25T18:00:58.495 INFO:teuthology.orchestra.run.smithi143.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:00:58.528 INFO:teuthology.orchestra.run.smithi143.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:00:58.587 INFO:teuthology.orchestra.run.smithi143.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:00:58.629 INFO:teuthology.orchestra.run.smithi143.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:00:58.662 INFO:teuthology.orchestra.run.smithi143.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:00:58.702 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:58.791 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:00:58.857 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:58.987 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:00:59.150 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:59.506 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:00:59.651 INFO:teuthology.orchestra.run.smithi203.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 203 kB/s | 498 kB 00:02 2024-03-25T18:00:59.992 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:00.318 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:00.353 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:00.398 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:00.496 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:00.522 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:00.582 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:00.743 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:01.111 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:01.246 INFO:teuthology.orchestra.run.smithi143.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 85 kB/s | 221 kB 00:02 2024-03-25T18:01:01.361 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:01.381 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:01.416 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:01.450 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:01.612 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:01.972 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:02.446 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:02.811 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:02.830 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:02.853 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:02.876 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:02.900 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:02.959 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:03.041 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:03.122 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:03.171 INFO:teuthology.orchestra.run.smithi143.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 19 kB/s | 36 kB 00:01 2024-03-25T18:01:03.192 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:03.213 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:03.246 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:03.280 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:03.444 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:03.799 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:04.270 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:04.598 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:04.615 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:04.640 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:04.665 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:04.683 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:04.753 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:04.834 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:04.915 INFO:teuthology.orchestra.run.smithi143.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:04.963 INFO:teuthology.orchestra.run.smithi143.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 26 kB/s | 47 kB 00:01 2024-03-25T18:01:05.055 INFO:teuthology.orchestra.run.smithi143.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.3 MB/s | 309 kB 00:00 2024-03-25T18:01:05.213 INFO:teuthology.orchestra.run.smithi143.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-03-25T18:01:05.247 INFO:teuthology.orchestra.run.smithi143.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 781 kB/s | 26 kB 00:00 2024-03-25T18:01:05.280 INFO:teuthology.orchestra.run.smithi143.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-25T18:01:05.314 INFO:teuthology.orchestra.run.smithi143.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-25T18:01:05.384 INFO:teuthology.orchestra.run.smithi143.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 20 MB/s | 838 kB 00:00 2024-03-25T18:01:05.398 INFO:teuthology.orchestra.run.smithi143.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-03-25T18:01:05.457 INFO:teuthology.orchestra.run.smithi143.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-03-25T18:01:11.517 INFO:teuthology.orchestra.run.smithi203.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 255 kB/s | 9.7 MB 00:39 2024-03-25T18:01:11.833 INFO:teuthology.orchestra.run.smithi203.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 142 kB/s | 45 kB 00:00 2024-03-25T18:01:12.714 INFO:teuthology.orchestra.run.smithi143.stdout:(29/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 250 kB/s | 9.7 MB 00:39 2024-03-25T18:01:12.918 INFO:teuthology.orchestra.run.smithi203.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 121 kB/s | 131 kB 00:01 2024-03-25T18:01:14.525 INFO:teuthology.orchestra.run.smithi203.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 101 kB/s | 162 kB 00:01 2024-03-25T18:01:16.000 INFO:teuthology.orchestra.run.smithi203.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 217 kB/s | 320 kB 00:01 2024-03-25T18:01:16.301 INFO:teuthology.orchestra.run.smithi203.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 314 kB/s | 5.1 MB 00:16 2024-03-25T18:01:16.626 INFO:teuthology.orchestra.run.smithi203.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 306 kB/s | 99 kB 00:00 2024-03-25T18:01:16.681 INFO:teuthology.orchestra.run.smithi143.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 308 kB/s | 3.4 MB 00:11 2024-03-25T18:01:16.734 INFO:teuthology.orchestra.run.smithi203.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-25T18:01:16.785 INFO:teuthology.orchestra.run.smithi203.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:01:16.818 INFO:teuthology.orchestra.run.smithi203.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:01:16.877 INFO:teuthology.orchestra.run.smithi203.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:01:16.919 INFO:teuthology.orchestra.run.smithi203.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:01:16.961 INFO:teuthology.orchestra.run.smithi203.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2024-03-25T18:01:17.030 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:17.166 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:17.332 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:17.685 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:17.730 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:17.826 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:17.861 INFO:teuthology.orchestra.run.smithi203.stdout:(18/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 160 kB/s | 298 kB 00:01 2024-03-25T18:01:17.893 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:17.895 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:17.953 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:18.018 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:18.034 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:18.053 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:18.095 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:18.114 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:18.163 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:18.186 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:18.258 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:18.325 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:18.332 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:18.350 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:18.357 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:18.374 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:18.395 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:18.415 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:18.453 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:18.472 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:18.828 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:18.949 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:19.153 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:19.217 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:19.236 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:19.291 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:19.351 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:19.368 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:19.410 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:19.529 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:19.763 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:19.781 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:19.836 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:19.877 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:19.894 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:19.961 INFO:teuthology.orchestra.run.smithi203.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 74 kB/s | 221 kB 00:02 2024-03-25T18:01:19.962 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:20.003 INFO:teuthology.orchestra.run.smithi203.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 17 kB/s | 36 kB 00:02 2024-03-25T18:01:20.037 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:20.175 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:01:20.334 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:20.395 INFO:teuthology.orchestra.run.smithi203.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 788 kB/s | 309 kB 00:00 2024-03-25T18:01:20.687 INFO:teuthology.orchestra.run.smithi203.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-25T18:01:20.711 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:01:20.754 INFO:teuthology.orchestra.run.smithi203.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-25T18:01:20.754 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:20.829 INFO:teuthology.orchestra.run.smithi203.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-25T18:01:20.854 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T18:01:20.896 INFO:teuthology.orchestra.run.smithi203.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T18:01:20.928 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:20.980 INFO:teuthology.orchestra.run.smithi203.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-25T18:01:21.072 INFO:teuthology.orchestra.run.smithi203.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T18:01:21.072 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:01:21.107 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:01:21.172 INFO:teuthology.orchestra.run.smithi203.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-25T18:01:21.172 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:21.257 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T18:01:21.388 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:21.408 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T18:01:21.878 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:22.231 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T18:01:22.283 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:22.309 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:22.470 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:22.817 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T18:01:22.838 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:22.889 INFO:teuthology.orchestra.run.smithi203.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:01:22.980 INFO:teuthology.orchestra.run.smithi203.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 16 kB/s | 47 kB 00:03 2024-03-25T18:01:26.665 INFO:teuthology.orchestra.run.smithi143.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 230 kB/s | 3.1 MB 00:13 2024-03-25T18:01:29.455 INFO:teuthology.orchestra.run.smithi203.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 417 kB/s | 3.4 MB 00:08 2024-03-25T18:01:34.021 INFO:teuthology.orchestra.run.smithi203.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 331 kB/s | 20 MB 01:01 2024-03-25T18:01:34.388 INFO:teuthology.orchestra.run.smithi203.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 281 kB/s | 3.1 MB 00:11 2024-03-25T18:01:34.395 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:34.395 INFO:teuthology.orchestra.run.smithi203.stdout:Total 950 kB/s | 58 MB 01:02 2024-03-25T18:01:34.428 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:01:34.467 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:01:34.467 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:01:34.938 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:01:34.938 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:01:36.400 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:01:36.485 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:01:36.517 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:01:36.549 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:01:36.574 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:36.591 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:01:36.783 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:36.847 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:01:37.174 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:01:37.236 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:37.328 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:37.415 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:01:37.437 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:37.508 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:01:37.537 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:01:37.586 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:01:37.649 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:37.673 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:37.720 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:01:37.755 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:01:37.863 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:01:37.906 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:01:37.947 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:01:37.979 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:01:38.008 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:01:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:01:38.355 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:01:38.387 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:38.477 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:01:38.508 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:01:38.553 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:01:39.006 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:01:39.032 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:39.094 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:01:39.163 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:01:39.192 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:01:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:41.447 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:41.999 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:01:42.042 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:42.134 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:01:42.134 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:01:42.134 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:01:42.134 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:01:42.185 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:51.668 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:51.669 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-03-25T18:01:51.669 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-03-25T18:01:51.669 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-03-25T18:01:51.669 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-03-25T18:01:51.669 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-03-25T18:01:51.669 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-03-25T18:01:51.669 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-03-25T18:01:51.669 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:01:52.589 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:01:52.613 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:01:52.613 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:01:52.613 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:01:52.613 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:01:52.614 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:01:52.614 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:01:52.616 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:01:52.692 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:01:52.692 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:53.891 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:53.891 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:01:53.892 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:01:53.893 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:01:53.894 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:01:53.894 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:01:53.894 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:01:53.894 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:01:53.894 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:01:53.894 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:01:53.894 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:01:53.894 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:01:54.876 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:01:54.876 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:01:54.876 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-03-25T18:01:54.876 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.876 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.876 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:01:54.877 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:01:54.878 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:01:54.879 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:01:54.879 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:01:54.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:01:54.879 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:01:55.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-03-25T18:01:55.760 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:26 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:01:55.909 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:01:55.910 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:01:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:01:55.910 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:01:55.910 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:01:55.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:01:55.910 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-03-25T18:01:55.911 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:01:55.912 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 48 M 2024-03-25T18:01:55.912 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 204 M 2024-03-25T18:01:55.912 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:01:56.189 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T18:01:56.231 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-25T18:01:56.264 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T18:01:56.806 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 345 kB/s | 187 kB 00:00 2024-03-25T18:01:56.939 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 179 kB/s | 163 kB 00:00 2024-03-25T18:01:59.042 INFO:teuthology.orchestra.run.smithi143.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 236 kB/s | 20 MB 01:26 2024-03-25T18:01:59.046 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:59.046 INFO:teuthology.orchestra.run.smithi143.stdout:Total 685 kB/s | 58 MB 01:26 2024-03-25T18:01:59.078 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:01:59.116 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:01:59.116 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:01:59.584 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:01:59.584 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:02:01.034 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:02:01.117 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:02:01.162 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:02:01.197 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:02:01.227 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:02:01.244 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:02:01.403 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:02:01.458 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:02:01.791 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:02:01.865 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:01.939 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:02.026 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:02:02.058 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:02:02.120 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:02:02.150 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:02:02.192 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:02:02.260 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:02.285 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:02:02.348 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:02:02.382 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:02:02.490 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:02:02.526 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:02:02.567 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:02:02.615 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:02:02.644 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:02:02.677 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:02:02.991 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:02:03.024 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:02:03.097 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:02:03.128 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:02:03.167 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:02:03.618 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:02:03.642 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:02:03.713 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:02:03.780 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:02:03.811 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:02:05.985 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:06.024 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:06.578 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:02:06.616 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:02:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:02:06.709 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:02:06.709 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:02:06.709 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:02:06.751 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:16.065 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:16.065 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /sys 2024-03-25T18:02:16.065 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /proc 2024-03-25T18:02:16.065 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /mnt 2024-03-25T18:02:16.065 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /var/tmp 2024-03-25T18:02:16.065 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /home 2024-03-25T18:02:16.066 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /root 2024-03-25T18:02:16.066 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /tmp 2024-03-25T18:02:16.066 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:02:16.980 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:17.007 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:02:17.007 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:02:17.007 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:02:17.007 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:17.007 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:02:17.007 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:02:17.009 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:17.091 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:02:17.091 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:02:18.229 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:02:18.230 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:02:18.231 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:02:18.231 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:02:18.231 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:02:18.231 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:02:18.231 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:02:18.232 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:02:18.232 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:02:18.232 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:02:18.232 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:02:18.232 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:02:19.401 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:02:19.401 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout:Upgraded: 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.402 INFO:teuthology.orchestra.run.smithi143.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.403 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:02:19.404 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:02:19.667 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-test 2024-03-25T18:02:20.221 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:50 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:02:20.372 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:02:20.372 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:02:20.373 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:02:20.374 INFO:teuthology.orchestra.run.smithi143.stdout:Install 6 Packages 2024-03-25T18:02:20.374 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:02:20.374 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 48 M 2024-03-25T18:02:20.374 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 204 M 2024-03-25T18:02:20.374 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:02:20.547 INFO:teuthology.orchestra.run.smithi143.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.9 MB/s | 217 kB 00:00 2024-03-25T18:02:20.647 INFO:teuthology.orchestra.run.smithi143.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.4 MB/s | 300 kB 00:00 2024-03-25T18:02:20.672 INFO:teuthology.orchestra.run.smithi143.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-25T18:02:21.181 INFO:teuthology.orchestra.run.smithi143.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 369 kB/s | 187 kB 00:00 2024-03-25T18:02:22.306 INFO:teuthology.orchestra.run.smithi143.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 87 kB/s | 163 kB 00:01 2024-03-25T18:04:28.895 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 317 kB/s | 47 MB 02:32 2024-03-25T18:04:28.897 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:28.897 INFO:teuthology.orchestra.run.smithi203.stdout:Total 322 kB/s | 48 MB 02:32 2024-03-25T18:04:28.908 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:04:28.928 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:04:28.928 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:04:29.309 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:04:29.310 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:04:29.898 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:04:29.969 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:04:30.008 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:04:30.049 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:04:30.097 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:04:30.122 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:04:34.596 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:04:34.672 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:04:37.089 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:04:37.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:04:37.089 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:04:37.090 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:04:37.090 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:04:37.090 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:04:37.431 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:04:37.431 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:04:37.431 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:04:37.431 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:37.431 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:04:37.432 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:37.432 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:04:37.432 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:04:37.432 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:04:37.432 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:04:37.432 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:04:37.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-03-25T18:04:38.332 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:04:09 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:04:38.489 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:04:38.492 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:04:38.493 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:04:38.494 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout:Install 36 Packages 2024-03-25T18:04:38.495 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:04:38.496 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 30 M 2024-03-25T18:04:38.496 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 107 M 2024-03-25T18:04:38.496 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:04:39.349 INFO:teuthology.orchestra.run.smithi203.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T18:04:47.507 INFO:teuthology.orchestra.run.smithi203.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 178 kB/s | 1.5 MB 00:08 2024-03-25T18:04:48.607 INFO:teuthology.orchestra.run.smithi203.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 225 kB/s | 2.1 MB 00:09 2024-03-25T18:04:49.741 INFO:teuthology.orchestra.run.smithi203.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 218 kB/s | 247 kB 00:01 2024-03-25T18:04:49.857 INFO:teuthology.orchestra.run.smithi203.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:04:49.899 INFO:teuthology.orchestra.run.smithi203.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:04:49.933 INFO:teuthology.orchestra.run.smithi203.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-03-25T18:04:49.967 INFO:teuthology.orchestra.run.smithi203.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T18:04:50.000 INFO:teuthology.orchestra.run.smithi203.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T18:04:50.042 INFO:teuthology.orchestra.run.smithi203.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:04:50.118 INFO:teuthology.orchestra.run.smithi203.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:04:50.160 INFO:teuthology.orchestra.run.smithi203.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T18:04:50.193 INFO:teuthology.orchestra.run.smithi203.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T18:04:50.235 INFO:teuthology.orchestra.run.smithi203.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:04:50.360 INFO:teuthology.orchestra.run.smithi203.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-25T18:04:50.986 INFO:teuthology.orchestra.run.smithi203.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-03-25T18:04:51.194 INFO:teuthology.orchestra.run.smithi203.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 727 kB/s | 151 kB 00:00 2024-03-25T18:04:51.253 INFO:teuthology.orchestra.run.smithi203.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 744 kB/s | 43 kB 00:00 2024-03-25T18:04:51.345 INFO:teuthology.orchestra.run.smithi203.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-25T18:04:51.446 INFO:teuthology.orchestra.run.smithi203.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T18:04:51.496 INFO:teuthology.orchestra.run.smithi203.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-25T18:04:51.554 INFO:teuthology.orchestra.run.smithi203.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-25T18:04:51.613 INFO:teuthology.orchestra.run.smithi203.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-03-25T18:04:51.664 INFO:teuthology.orchestra.run.smithi203.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-03-25T18:04:51.722 INFO:teuthology.orchestra.run.smithi203.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-25T18:04:51.773 INFO:teuthology.orchestra.run.smithi203.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-03-25T18:04:51.831 INFO:teuthology.orchestra.run.smithi203.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T18:04:51.898 INFO:teuthology.orchestra.run.smithi203.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T18:04:51.949 INFO:teuthology.orchestra.run.smithi203.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-03-25T18:04:52.007 INFO:teuthology.orchestra.run.smithi203.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T18:04:52.058 INFO:teuthology.orchestra.run.smithi203.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-25T18:04:52.125 INFO:teuthology.orchestra.run.smithi203.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T18:04:52.209 INFO:teuthology.orchestra.run.smithi203.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-25T18:04:52.259 INFO:teuthology.orchestra.run.smithi203.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-25T18:04:59.283 INFO:teuthology.orchestra.run.smithi203.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 241 kB/s | 4.7 MB 00:19 2024-03-25T18:05:07.291 INFO:teuthology.orchestra.run.smithi143.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 290 kB/s | 47 MB 02:46 2024-03-25T18:05:07.294 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:07.294 INFO:teuthology.orchestra.run.smithi143.stdout:Total 296 kB/s | 48 MB 02:46 2024-03-25T18:05:07.305 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:05:07.325 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:05:07.325 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:05:07.701 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:05:07.701 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:05:08.304 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:05:08.383 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:05:08.420 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:05:08.451 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:05:08.495 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:05:08.520 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:05:12.991 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:05:13.069 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:05:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:05:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:05:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:05:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:05:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:05:15.510 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:05:15.935 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:05:16.238 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph 2024-03-25T18:05:16.801 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:05:16.964 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout:========================================================================================= 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout:========================================================================================= 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:05:16.967 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:05:16.968 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:05:16.969 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:05:16.970 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:05:16.970 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:05:16.970 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:05:16.970 INFO:teuthology.orchestra.run.smithi143.stdout:========================================================================================= 2024-03-25T18:05:16.970 INFO:teuthology.orchestra.run.smithi143.stdout:Install 36 Packages 2024-03-25T18:05:16.970 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:05:16.971 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 30 M 2024-03-25T18:05:16.971 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 107 M 2024-03-25T18:05:16.971 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:05:17.863 INFO:teuthology.orchestra.run.smithi143.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:05:21.154 INFO:teuthology.orchestra.run.smithi143.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 408 kB/s | 1.5 MB 00:03 2024-03-25T18:05:27.129 INFO:teuthology.orchestra.run.smithi143.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 224 kB/s | 2.1 MB 00:09 2024-03-25T18:05:29.062 INFO:teuthology.orchestra.run.smithi143.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 128 kB/s | 247 kB 00:01 2024-03-25T18:05:29.179 INFO:teuthology.orchestra.run.smithi143.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:05:29.221 INFO:teuthology.orchestra.run.smithi143.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:05:29.246 INFO:teuthology.orchestra.run.smithi143.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:05:29.272 INFO:teuthology.orchestra.run.smithi143.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-25T18:05:29.297 INFO:teuthology.orchestra.run.smithi143.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:05:29.339 INFO:teuthology.orchestra.run.smithi143.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T18:05:29.406 INFO:teuthology.orchestra.run.smithi143.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T18:05:29.439 INFO:teuthology.orchestra.run.smithi143.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:05:29.473 INFO:teuthology.orchestra.run.smithi143.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:05:29.506 INFO:teuthology.orchestra.run.smithi143.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T18:05:29.640 INFO:teuthology.orchestra.run.smithi143.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 938 kB/s | 125 kB 00:00 2024-03-25T18:05:29.907 INFO:teuthology.orchestra.run.smithi143.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-25T18:05:30.166 INFO:teuthology.orchestra.run.smithi143.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 585 kB/s | 151 kB 00:00 2024-03-25T18:05:30.233 INFO:teuthology.orchestra.run.smithi143.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 651 kB/s | 43 kB 00:00 2024-03-25T18:05:30.358 INFO:teuthology.orchestra.run.smithi143.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-25T18:05:30.492 INFO:teuthology.orchestra.run.smithi143.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-25T18:05:30.575 INFO:teuthology.orchestra.run.smithi143.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-25T18:05:30.642 INFO:teuthology.orchestra.run.smithi143.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-25T18:05:30.709 INFO:teuthology.orchestra.run.smithi143.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T18:05:30.776 INFO:teuthology.orchestra.run.smithi143.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-25T18:05:30.852 INFO:teuthology.orchestra.run.smithi143.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-25T18:05:30.935 INFO:teuthology.orchestra.run.smithi143.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 555 kB/s | 46 kB 00:00 2024-03-25T18:05:31.010 INFO:teuthology.orchestra.run.smithi143.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T18:05:31.144 INFO:teuthology.orchestra.run.smithi143.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-25T18:05:31.211 INFO:teuthology.orchestra.run.smithi143.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-03-25T18:05:31.295 INFO:teuthology.orchestra.run.smithi143.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-25T18:05:31.370 INFO:teuthology.orchestra.run.smithi143.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-03-25T18:05:31.445 INFO:teuthology.orchestra.run.smithi143.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-25T18:05:31.513 INFO:teuthology.orchestra.run.smithi143.stdout:(33/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 351 kB/s | 4.7 MB 00:13 2024-03-25T18:05:31.605 INFO:teuthology.orchestra.run.smithi143.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-25T18:05:31.705 INFO:teuthology.orchestra.run.smithi143.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 104 kB/s | 20 kB 00:00 2024-03-25T18:05:45.577 INFO:teuthology.orchestra.run.smithi203.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 296 kB/s | 17 MB 00:58 2024-03-25T18:05:45.581 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:45.581 INFO:teuthology.orchestra.run.smithi203.stdout:Total 453 kB/s | 30 MB 01:07 2024-03-25T18:05:45.602 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:05:45.647 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:05:45.647 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:05:46.061 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:05:46.061 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:05:47.417 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:05:47.471 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:05:47.521 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:05:47.548 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:05:47.619 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:05:47.656 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:05:47.689 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:05:47.780 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:05:47.858 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:05:47.900 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:05:47.971 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:05:48.025 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:05:48.064 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:05:48.113 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:05:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:05:48.353 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:05:48.402 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:05:48.432 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:05:48.467 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:05:48.497 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:05:48.574 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:05:48.608 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:48.641 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:05:48.681 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:05:48.716 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:05:48.751 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:05:48.832 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:05:48.874 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:05:48.946 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:05:49.083 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:05:49.210 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:49.365 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:05:49.388 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:49.412 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:49.413 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:49.413 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:05:49.413 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:49.413 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:49.413 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:05:50.864 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:05:50.894 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:05:50.894 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:50.894 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:05:50.894 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:05:50.895 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:05:50.895 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:05:51.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:05:51.382 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:05:51.382 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:51.382 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:05:51.382 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:05:51.382 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:05:51.383 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:05:51.600 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:05:51.622 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:05:51.623 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:51.623 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:05:51.623 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:05:51.623 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:05:51.623 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:05:51.691 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:05:53.224 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:05:53.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:05:53.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:05:53.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:05:53.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:05:53.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:05:53.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:05:53.225 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:53.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:05:53.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:05:53.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:05:53.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:05:53.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:05:53.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:05:53.226 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:05:53.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:05:53.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:53.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:05:53.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:05:53.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:05:53.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:05:53.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:05:54.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:05:54.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:05:54.329 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:05:54.329 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:05:54.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-03-25T18:05:55.247 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:05:55.344 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:05:55.396 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:05:55.397 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:05:55.397 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:05:55.478 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-03-25T18:05:56.041 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:27 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:05:56.135 INFO:teuthology.orchestra.run.smithi143.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 491 kB/s | 17 MB 00:34 2024-03-25T18:05:56.139 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:56.139 INFO:teuthology.orchestra.run.smithi143.stdout:Total 776 kB/s | 30 MB 00:39 2024-03-25T18:05:56.160 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:05:56.188 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:05:56.189 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:05:56.189 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:05:56.189 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:05:56.189 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:05:56.189 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:05:56.190 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:05:56.190 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:05:56.190 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:05:56.190 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:05:56.190 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:05:56.190 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 747 k 2024-03-25T18:05:56.190 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 754 k 2024-03-25T18:05:56.190 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:05:56.201 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:05:56.201 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:05:56.600 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:05:56.600 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:05:57.914 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:05:57.979 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:05:58.024 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:05:58.056 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:05:58.126 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:05:58.172 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:05:58.205 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:05:58.294 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:05:58.366 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:05:58.407 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:05:58.479 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:05:58.533 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:05:58.571 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:05:58.621 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:05:58.675 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:05:58.860 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:05:58.909 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:05:58.940 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:05:58.974 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:05:59.004 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:05:59.081 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:05:59.109 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 256 kB/s | 747 kB 00:02 2024-03-25T18:05:59.110 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:59.110 INFO:teuthology.orchestra.run.smithi203.stdout:Total 256 kB/s | 747 kB 00:02 2024-03-25T18:05:59.110 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:05:59.115 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:05:59.117 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:05:59.117 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:05:59.123 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:05:59.123 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:05:59.148 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:05:59.188 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:05:59.209 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:05:59.223 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:05:59.266 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:05:59.355 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:05:59.402 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:05:59.475 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:05:59.610 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:05:59.637 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:59.713 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:05:59.737 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:05:59.903 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:05:59.932 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:59.957 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:05:59.958 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:05:59.958 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:05:59.958 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:59.958 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:05:59.958 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:00.109 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:00.386 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:00.387 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:00.387 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:00.387 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:00.387 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:00.387 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:00.613 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:06:01.183 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:01.336 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:01.337 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:06:01.338 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:01.338 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:06:01.338 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:01.338 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 145 k 2024-03-25T18:06:01.338 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 436 k 2024-03-25T18:06:01.338 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:06:01.435 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:06:01.463 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:06:01.463 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:01.463 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:06:01.463 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:06:01.463 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:06:01.463 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:01.911 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:06:01.934 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:06:01.934 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:01.934 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:06:01.934 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:06:01.934 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:06:01.934 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:02.136 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:06:02.159 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:06:02.159 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:02.159 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:06:02.160 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:06:02.160 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:06:02.160 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:02.226 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:06:02.666 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 109 kB/s | 145 kB 00:01 2024-03-25T18:06:02.667 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:02.668 INFO:teuthology.orchestra.run.smithi203.stdout:Total 109 kB/s | 145 kB 00:01 2024-03-25T18:06:02.668 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:06:02.677 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:06:02.677 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:06:02.734 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:06:02.735 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:06:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:06:02.950 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:02.974 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:02.974 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:02.974 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:06:02.975 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:03.773 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:06:03.773 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:06:03.773 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:06:03.773 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:06:03.773 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:06:03.773 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:06:03.773 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:06:03.774 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:06:03.775 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:06:03.776 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:06:03.776 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:06:03.776 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:06:03.776 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:06:03.776 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:06:03.776 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:06:03.936 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:03.936 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:03.936 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:03.936 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:03.936 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:03.936 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:04.215 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-03-25T18:06:04.788 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:04.887 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:04.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:06:04.906 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:04.906 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:06:04.906 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:06:04.907 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:06:04.908 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:06:04.909 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:06:04.910 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:06:04.910 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:06:04.910 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:06:04.910 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:04.910 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:06:04.939 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:04.940 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:06:04.940 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:05.017 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:06:05.298 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-base 2024-03-25T18:06:05.575 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:05.724 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:05.725 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-25T18:06:05.725 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:06:05.725 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-25T18:06:05.725 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:06:05.725 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:06:05.725 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:06:05.726 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-25T18:06:05.727 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-03-25T18:06:05.727 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:05.727 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.9 M 2024-03-25T18:06:05.727 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-03-25T18:06:05.727 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:06:05.881 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:06:05.979 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:06.031 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:06:06.032 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:06:06.032 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:06:06.115 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install cephadm 2024-03-25T18:06:06.380 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 35 kB/s | 15 kB 00:00 2024-03-25T18:06:06.530 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 53 kB/s | 31 kB 00:00 2024-03-25T18:06:06.672 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-03-25T18:06:06.674 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:06:06.697 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-03-25T18:06:06.824 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:06:06.825 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:06:06.826 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:06.826 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 747 k 2024-03-25T18:06:06.826 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 754 k 2024-03-25T18:06:06.826 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:06:06.964 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.0 MB/s | 548 kB 00:00 2024-03-25T18:06:07.006 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-03-25T18:06:07.032 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-03-25T18:06:07.157 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-03-25T18:06:07.199 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 739 kB/s | 31 kB 00:00 2024-03-25T18:06:07.241 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 691 kB/s | 144 kB 00:00 2024-03-25T18:06:07.266 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.7 MB/s | 188 kB 00:00 2024-03-25T18:06:09.329 INFO:teuthology.orchestra.run.smithi143.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 299 kB/s | 747 kB 00:02 2024-03-25T18:06:09.329 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:09.330 INFO:teuthology.orchestra.run.smithi143.stdout:Total 298 kB/s | 747 kB 00:02 2024-03-25T18:06:09.330 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:06:09.336 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:06:09.336 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:06:09.342 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:06:09.342 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:06:09.366 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 532 kB/s | 1.8 MB 00:03 2024-03-25T18:06:09.370 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:09.370 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.9 MB/s | 6.9 MB 00:03 2024-03-25T18:06:09.389 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:06:09.408 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:06:09.408 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:06:09.413 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:06:09.587 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:06:09.588 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:06:09.873 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:09.967 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:10.105 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:06:10.183 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:06:10.227 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:06:10.281 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:06:10.321 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:06:10.351 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:10.416 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:06:10.546 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:06:10.632 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:06:10.644 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:06:10.644 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:10.644 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:06:10.644 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:10.644 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:10.644 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:06:10.676 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:06:10.713 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:06:10.745 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:06:10.850 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:06:10.987 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:06:11.020 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:06:11.421 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:06:11.583 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:06:11.584 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:06:11.584 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:06:11.584 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:06:11.584 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:06:11.584 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:06:11.584 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:11.585 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:06:11.585 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:06:11.585 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:06:11.585 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:11.585 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 145 k 2024-03-25T18:06:11.585 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 436 k 2024-03-25T18:06:11.585 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:06:11.860 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:06:11.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:06:11.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:06:11.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:06:11.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:06:11.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:06:11.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:06:11.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:06:11.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:06:11.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:06:11.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:06:11.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:06:12.200 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:06:12.200 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:12.200 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:12.200 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:12.201 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:12.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:06:13.030 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:13.179 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:13.181 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-25T18:06:13.181 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:06:13.181 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-25T18:06:13.181 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:06:13.181 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:06:13.181 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:06:13.181 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-03-25T18:06:13.182 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:13.183 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 37 M 2024-03-25T18:06:13.183 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 209 M 2024-03-25T18:06:13.183 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:06:13.319 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 84 kB/s | 145 kB 00:01 2024-03-25T18:06:13.319 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:13.320 INFO:teuthology.orchestra.run.smithi143.stdout:Total 84 kB/s | 145 kB 00:01 2024-03-25T18:06:13.320 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:06:13.329 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:06:13.329 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:06:13.384 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:06:13.385 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:06:13.528 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:06:13.594 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:13.616 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 260 kB/s | 32 kB 00:00 2024-03-25T18:06:13.618 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:13.619 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:13.619 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:06:13.619 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:13.675 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 487 kB/s | 17 kB 00:00 2024-03-25T18:06:13.717 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 818 kB/s | 34 kB 00:00 2024-03-25T18:06:14.067 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-25T18:06:14.242 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-25T18:06:14.301 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 217 kB 00:00 2024-03-25T18:06:14.535 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-03-25T18:06:14.601 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:06:14.601 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:14.601 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:06:14.601 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:14.601 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:14.601 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:06:14.843 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-25T18:06:14.847 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr 2024-03-25T18:06:15.035 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-25T18:06:15.110 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-25T18:06:15.421 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:06:15.518 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:06:15.570 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:06:15.571 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:06:15.571 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:06:15.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:06:15.977 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 260 kB/s | 245 kB 00:00 2024-03-25T18:06:16.216 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:06:16.366 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout:====================================================================================== 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout:====================================================================================== 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:06:16.367 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout:====================================================================================== 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout:Install 12 Packages 2024-03-25T18:06:16.368 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:16.369 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 6.9 M 2024-03-25T18:06:16.369 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 26 M 2024-03-25T18:06:16.369 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:06:16.940 INFO:teuthology.orchestra.run.smithi143.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 69 kB/s | 31 kB 00:00 2024-03-25T18:06:17.040 INFO:teuthology.orchestra.run.smithi143.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 28 kB/s | 15 kB 00:00 2024-03-25T18:06:17.098 INFO:teuthology.orchestra.run.smithi143.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.3 MB/s | 1.0 MB 00:00 2024-03-25T18:06:17.124 INFO:teuthology.orchestra.run.smithi143.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T18:06:17.391 INFO:teuthology.orchestra.run.smithi143.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.9 MB/s | 863 kB 00:00 2024-03-25T18:06:17.649 INFO:teuthology.orchestra.run.smithi143.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-25T18:06:17.774 INFO:teuthology.orchestra.run.smithi143.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 843 kB/s | 548 kB 00:00 2024-03-25T18:06:17.950 INFO:teuthology.orchestra.run.smithi143.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 824 kB/s | 144 kB 00:00 2024-03-25T18:06:18.108 INFO:teuthology.orchestra.run.smithi143.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 194 kB/s | 31 kB 00:00 2024-03-25T18:06:18.259 INFO:teuthology.orchestra.run.smithi143.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-03-25T18:06:18.601 INFO:teuthology.orchestra.run.smithi143.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.1 MB/s | 2.0 MB 00:00 2024-03-25T18:06:21.526 INFO:teuthology.orchestra.run.smithi143.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 362 kB/s | 1.8 MB 00:05 2024-03-25T18:06:21.530 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:21.530 INFO:teuthology.orchestra.run.smithi143.stdout:Total 1.3 MB/s | 6.9 MB 00:05 2024-03-25T18:06:21.550 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:06:21.568 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:06:21.569 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:06:21.760 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:06:21.760 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:06:22.288 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:06:22.376 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:06:22.411 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:06:22.458 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:06:22.488 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:06:22.591 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:06:22.713 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:06:22.807 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:06:22.851 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:06:22.880 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:06:22.911 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:06:23.153 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:06:23.195 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:06:23.988 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:06:23.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:06:24.342 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:06:24.343 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:06:24.343 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:06:24.343 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:06:24.343 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:24.343 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:06:24.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:06:25.190 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:06:25.341 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:06:25.342 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-25T18:06:25.342 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:06:25.342 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:06:25.343 INFO:teuthology.orchestra.run.smithi143.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout:Install 12 Packages 2024-03-25T18:06:25.344 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:25.345 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 37 M 2024-03-25T18:06:25.345 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 209 M 2024-03-25T18:06:25.345 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:06:25.632 INFO:teuthology.orchestra.run.smithi143.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 285 kB/s | 32 kB 00:00 2024-03-25T18:06:25.699 INFO:teuthology.orchestra.run.smithi143.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 395 kB/s | 17 kB 00:00 2024-03-25T18:06:25.733 INFO:teuthology.orchestra.run.smithi143.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:06:25.891 INFO:teuthology.orchestra.run.smithi143.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T18:06:26.025 INFO:teuthology.orchestra.run.smithi143.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T18:06:26.100 INFO:teuthology.orchestra.run.smithi143.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-25T18:06:26.125 INFO:teuthology.orchestra.run.smithi143.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-03-25T18:06:26.301 INFO:teuthology.orchestra.run.smithi143.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T18:06:26.460 INFO:teuthology.orchestra.run.smithi143.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-25T18:06:26.485 INFO:teuthology.orchestra.run.smithi143.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-25T18:06:26.710 INFO:teuthology.orchestra.run.smithi143.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 982 kB/s | 245 kB 00:00 2024-03-25T18:06:29.242 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 480 kB/s | 7.4 MB 00:15 2024-03-25T18:06:29.245 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:29.246 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.3 MB/s | 37 MB 00:16 2024-03-25T18:06:29.289 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:06:29.315 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:06:29.315 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:06:29.676 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:06:29.676 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:06:30.400 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:06:30.466 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:06:30.497 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:06:30.588 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:06:30.642 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:06:30.823 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:06:30.849 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:06:30.933 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:06:31.578 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:06:31.698 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:06:32.936 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:06:33.975 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:06:34.011 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:06:36.662 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:06:36.662 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:06:36.662 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:06:36.662 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:06:36.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:06:36.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:06:36.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:06:36.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:06:36.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:06:36.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:06:36.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:06:36.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:06:37.091 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:06:37.091 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:37.091 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:37.091 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:37.092 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:37.498 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:06:38.072 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:09 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:38.224 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:38.225 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:06:38.225 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:06:38.225 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:06:38.226 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-03-25T18:06:38.227 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:38.228 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2024-03-25T18:06:38.228 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-03-25T18:06:38.228 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:06:38.573 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 169 kB/s | 24 kB 00:00 2024-03-25T18:06:38.589 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2024-03-25T18:06:38.656 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:06:38.682 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-25T18:06:38.707 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-25T18:06:38.724 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T18:06:38.774 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-03-25T18:06:38.791 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 223 kB/s | 15 kB 00:00 2024-03-25T18:06:38.850 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.6 MB/s | 203 kB 00:00 2024-03-25T18:06:38.883 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-03-25T18:06:38.934 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-03-25T18:06:38.950 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T18:06:39.151 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 72 kB/s | 52 kB 00:00 2024-03-25T18:06:39.153 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:39.153 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T18:06:39.153 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:06:39.166 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:06:39.166 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:06:39.255 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:06:39.255 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:06:39.769 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:06:39.924 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:06:39.977 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:06:40.041 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:06:40.079 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:06:40.120 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:06:40.214 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:06:40.315 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:06:40.380 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:06:41.105 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:06:41.161 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:06:41.195 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:06:41.243 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:06:41.265 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:06:42.039 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:06:42.039 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:06:42.039 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:06:42.039 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:06:42.039 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:06:42.040 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:06:42.040 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:06:42.040 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:06:42.040 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:06:42.040 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:06:42.040 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:06:42.040 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:06:42.040 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:06:42.200 INFO:teuthology.orchestra.run.smithi143.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 454 kB/s | 7.4 MB 00:16 2024-03-25T18:06:42.202 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:42.202 INFO:teuthology.orchestra.run.smithi143.stdout:Total 2.2 MB/s | 37 MB 00:16 2024-03-25T18:06:42.246 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:06:42.272 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:06:42.272 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:06:42.398 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:42.399 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:42.631 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:06:42.632 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:06:42.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:06:43.236 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:43.335 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:06:43.385 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:43.386 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:06:43.386 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-03-25T18:06:43.387 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:43.388 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.8 M 2024-03-25T18:06:43.388 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 33 M 2024-03-25T18:06:43.388 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:06:43.397 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:06:43.438 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:06:43.521 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:06:43.557 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:06:43.734 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:06:43.764 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:06:43.850 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:06:43.864 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T18:06:44.497 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:06:44.531 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 853 kB/s | 548 kB 00:00 2024-03-25T18:06:44.565 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 257 kB/s | 248 kB 00:00 2024-03-25T18:06:44.613 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:06:44.632 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 575 kB/s | 58 kB 00:00 2024-03-25T18:06:44.748 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 120 kB/s | 138 kB 00:01 2024-03-25T18:06:44.899 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 153 kB/s | 51 kB 00:00 2024-03-25T18:06:44.900 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:44.901 INFO:teuthology.orchestra.run.smithi203.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-03-25T18:06:44.914 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:06:44.924 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:06:44.924 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:06:45.017 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:06:45.018 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:06:45.334 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:06:45.508 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:06:45.568 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:06:45.819 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:06:46.092 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:06:46.178 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:06:46.249 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:06:46.272 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:06:46.833 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:06:46.857 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:06:47.083 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:06:47.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:06:47.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:06:47.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:06:47.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:06:47.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:06:47.377 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:06:47.377 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:47.377 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:47.377 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:47.377 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:06:47.377 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:06:47.377 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:06:47.378 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:06:47.378 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:06:47.378 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:47.378 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:47.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-03-25T18:06:48.187 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:19 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:48.337 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:48.337 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:48.341 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 964 k 2024-03-25T18:06:48.342 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.8 M 2024-03-25T18:06:48.342 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:06:48.460 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 650 kB/s | 79 kB 00:00 2024-03-25T18:06:49.379 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:06:49.379 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:06:49.379 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:06:49.379 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:06:49.379 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:06:49.380 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:06:49.380 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:06:49.380 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:06:49.380 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:06:49.380 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:06:49.380 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:06:49.380 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:06:49.767 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:06:49.767 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:49.767 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:06:49.767 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:49.767 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:06:49.767 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:49.768 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:06:50.156 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:06:50.733 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:20 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:06:50.827 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 356 kB/s | 885 kB 00:02 2024-03-25T18:06:50.828 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:50.829 INFO:teuthology.orchestra.run.smithi203.stdout:Total 387 kB/s | 964 kB 00:02 2024-03-25T18:06:50.829 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:06:50.854 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:06:50.854 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:06:50.884 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:06:50.885 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:06:50.885 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:06:50.886 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:06:50.887 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:06:50.887 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:06:50.887 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:06:50.887 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:50.887 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:06:50.887 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:06:50.887 INFO:teuthology.orchestra.run.smithi143.stdout:Install 13 Packages 2024-03-25T18:06:50.887 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:50.888 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 2.0 M 2024-03-25T18:06:50.888 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 26 M 2024-03-25T18:06:50.888 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:06:50.931 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:06:50.931 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:06:51.066 INFO:teuthology.orchestra.run.smithi143.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 18 kB 00:00 2024-03-25T18:06:51.091 INFO:teuthology.orchestra.run.smithi143.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 153 kB/s | 24 kB 00:00 2024-03-25T18:06:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:06:51.183 INFO:teuthology.orchestra.run.smithi143.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 174 kB 00:00 2024-03-25T18:06:51.208 INFO:teuthology.orchestra.run.smithi143.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 132 kB 00:00 2024-03-25T18:06:51.261 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:06:51.284 INFO:teuthology.orchestra.run.smithi143.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.1 MB/s | 211 kB 00:00 2024-03-25T18:06:51.309 INFO:teuthology.orchestra.run.smithi143.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 427 kB/s | 43 kB 00:00 2024-03-25T18:06:51.327 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:51.409 INFO:teuthology.orchestra.run.smithi143.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 150 kB/s | 15 kB 00:00 2024-03-25T18:06:51.435 INFO:teuthology.orchestra.run.smithi143.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-03-25T18:06:51.610 INFO:teuthology.orchestra.run.smithi143.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 77 kB/s | 52 kB 00:00 2024-03-25T18:06:51.635 INFO:teuthology.orchestra.run.smithi143.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 899 kB/s | 203 kB 00:00 2024-03-25T18:06:51.702 INFO:teuthology.orchestra.run.smithi143.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T18:06:51.727 INFO:teuthology.orchestra.run.smithi143.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-03-25T18:06:51.861 INFO:teuthology.orchestra.run.smithi143.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-03-25T18:06:51.862 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:51.863 INFO:teuthology.orchestra.run.smithi143.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-25T18:06:51.863 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:06:51.966 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:06:51.966 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:06:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:06:52.015 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:06:52.327 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:06:52.328 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:52.328 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:52.328 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:06:52.328 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:52.328 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:52.490 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:06:52.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-volume 2024-03-25T18:06:52.656 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:06:52.704 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:06:52.767 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:06:52.805 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:06:52.846 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:06:52.939 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:06:53.039 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:06:53.097 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:06:53.182 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:24 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:53.335 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:53.336 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:53.336 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-25T18:06:53.336 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:53.336 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:06:53.336 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:06:53.336 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:06:53.336 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:06:53.336 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:53.337 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:06:53.337 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:53.337 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-03-25T18:06:53.337 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:53.337 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 332 k 2024-03-25T18:06:53.338 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.5 M 2024-03-25T18:06:53.338 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:06:53.450 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 611 kB/s | 69 kB 00:00 2024-03-25T18:06:53.814 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:06:53.864 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:06:53.897 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:06:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:06:53.967 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:06:54.675 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 197 kB/s | 264 kB 00:01 2024-03-25T18:06:54.676 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:54.677 INFO:teuthology.orchestra.run.smithi203.stdout:Total 248 kB/s | 332 kB 00:01 2024-03-25T18:06:54.677 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:06:54.687 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:06:54.687 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:06:54.729 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:06:54.730 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:06:54.799 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:06:54.799 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:06:54.799 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:06:54.799 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:06:54.799 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:06:54.799 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:06:54.799 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:06:54.800 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:06:54.800 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:06:54.800 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:06:54.800 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:06:54.800 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:06:54.800 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:06:54.879 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:06:55.069 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:06:55.119 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:55.135 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:06:55.135 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:06:55.135 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:06:55.135 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:55.171 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:06:55.171 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:55.171 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:06:55.171 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:55.171 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:06:55.171 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:06:55.171 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:06:55.171 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:55.172 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:06:55.435 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:06:55.888 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:06:55.997 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:25 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:06:56.148 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:06:56.149 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:56.150 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:06:56.150 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:06:56.150 INFO:teuthology.orchestra.run.smithi143.stdout:Install 6 Packages 2024-03-25T18:06:56.150 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:06:56.150 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 6.8 M 2024-03-25T18:06:56.150 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 33 M 2024-03-25T18:06:56.150 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:06:56.222 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:06:56.222 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:56.222 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:56.222 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:06:56.222 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:06:56.222 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:56.222 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:56.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-03-25T18:06:56.807 INFO:teuthology.orchestra.run.smithi143.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-25T18:06:57.170 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:28 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:06:57.182 INFO:teuthology.orchestra.run.smithi143.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T18:06:57.265 INFO:teuthology.orchestra.run.smithi143.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 695 kB/s | 58 kB 00:00 2024-03-25T18:06:57.319 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:06:57.319 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:06:57.320 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:57.321 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 127 k 2024-03-25T18:06:57.321 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 456 k 2024-03-25T18:06:57.321 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:06:57.349 INFO:teuthology.orchestra.run.smithi143.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 611 kB/s | 51 kB 00:00 2024-03-25T18:06:57.574 INFO:teuthology.orchestra.run.smithi143.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 230 kB/s | 248 kB 00:01 2024-03-25T18:06:57.658 INFO:teuthology.orchestra.run.smithi143.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 119 kB/s | 138 kB 00:01 2024-03-25T18:06:57.660 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:57.660 INFO:teuthology.orchestra.run.smithi143.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2024-03-25T18:06:57.673 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:06:57.684 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:06:57.684 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:06:57.772 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:06:57.772 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:06:58.065 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:06:58.223 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:06:58.267 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:06:58.624 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 97 kB/s | 127 kB 00:01 2024-03-25T18:06:58.624 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:58.625 INFO:teuthology.orchestra.run.smithi203.stdout:Total 97 kB/s | 127 kB 00:01 2024-03-25T18:06:58.625 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:06:58.631 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:06:58.632 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:06:58.691 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:06:58.691 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:06:58.786 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:06:58.839 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:06:58.870 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:06:58.932 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:06:58.940 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:58.955 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:06:59.414 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:59.664 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:59.665 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:06:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:59.665 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:06:59.665 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:06:59.780 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:06:59.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:00.087 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:00.333 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-fuse 2024-03-25T18:07:00.485 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:31 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:00.583 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:00.637 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:00.638 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:07:00.638 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:00.717 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-03-25T18:07:00.893 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:01.042 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:07:01.043 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:01.044 INFO:teuthology.orchestra.run.smithi143.stdout:Install 2 Packages 2024-03-25T18:07:01.044 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:01.044 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 964 k 2024-03-25T18:07:01.044 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 2.8 M 2024-03-25T18:07:01.044 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:07:01.143 INFO:teuthology.orchestra.run.smithi143.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 795 kB/s | 79 kB 00:00 2024-03-25T18:07:01.280 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:32 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:01.429 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:01.429 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:07:01.430 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:01.431 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 31 k 2024-03-25T18:07:01.431 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 140 k 2024-03-25T18:07:01.431 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:07:01.963 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 59 kB/s | 31 kB 00:00 2024-03-25T18:07:01.963 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:01.963 INFO:teuthology.orchestra.run.smithi203.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T18:07:01.963 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:07:01.967 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:07:01.967 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:07:01.994 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:07:01.994 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:07:02.123 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:07:02.236 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:02.613 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:02.885 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:02.885 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:02.885 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:07:02.885 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:02.885 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:02.885 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:03.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-03-25T18:07:03.611 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:34 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:03.708 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:03.759 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:03.760 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:07:03.760 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:03.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-03-25T18:07:04.382 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:35 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:04.482 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:04.536 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:04.537 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:07:04.537 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:04.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-03-25T18:07:04.985 INFO:teuthology.orchestra.run.smithi143.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 225 kB/s | 885 kB 00:03 2024-03-25T18:07:04.987 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:04.987 INFO:teuthology.orchestra.run.smithi143.stdout:Total 244 kB/s | 964 kB 00:03 2024-03-25T18:07:04.987 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:07:05.010 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:07:05.010 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:07:05.090 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:07:05.091 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:07:05.199 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:36 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:05.267 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:07:05.296 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:05.349 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:05.350 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:07:05.350 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:05.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-03-25T18:07:05.402 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:07:05.467 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:07:05.966 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:36 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:06.063 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:06.114 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:06.115 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:07:06.115 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:06.124 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:07:06.124 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:07:06.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-03-25T18:07:06.443 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:07:06.443 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:06.443 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:07:06.443 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:07:06.443 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:06.443 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:06.731 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:37 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:06.739 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-volume 2024-03-25T18:07:06.829 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:06.881 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:06.882 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:07:06.882 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:06.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-03-25T18:07:07.317 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:37 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:07.473 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:07.473 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:07.473 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-25T18:07:07.473 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout:Install 2 Packages 2024-03-25T18:07:07.474 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:07.475 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 332 k 2024-03-25T18:07:07.475 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 1.5 M 2024-03-25T18:07:07.475 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:07:07.502 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:38 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:07.582 INFO:teuthology.orchestra.run.smithi143.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 644 kB/s | 69 kB 00:00 2024-03-25T18:07:07.599 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:07.652 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:07.653 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:07:07.653 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:07.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-03-25T18:07:08.273 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:39 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:08.424 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:08.424 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:08.424 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-03-25T18:07:08.424 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 85 k 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 231 k 2024-03-25T18:07:08.425 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:07:09.208 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 109 kB/s | 85 kB 00:00 2024-03-25T18:07:09.209 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:09.209 INFO:teuthology.orchestra.run.smithi203.stdout:Total 109 kB/s | 85 kB 00:00 2024-03-25T18:07:09.209 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:07:09.215 INFO:teuthology.orchestra.run.smithi143.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 152 kB/s | 264 kB 00:01 2024-03-25T18:07:09.216 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:09.217 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:07:09.217 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:07:09.217 INFO:teuthology.orchestra.run.smithi143.stdout:Total 191 kB/s | 332 kB 00:01 2024-03-25T18:07:09.217 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:07:09.227 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:07:09.227 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:07:09.269 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:07:09.269 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:07:09.277 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:07:09.277 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:07:09.414 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:07:09.420 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:07:09.515 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:09.600 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:07:09.642 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:07:09.658 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:07:09.659 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:07:09.659 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:07:09.659 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:09.953 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:10.240 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:10.240 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:10.240 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:07:10.240 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:10.240 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:10.240 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:10.395 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:07:10.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-03-25T18:07:10.702 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:07:10.703 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:10.703 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:07:10.703 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:07:10.703 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:07:10.703 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:10.703 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:11.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados-devel 2024-03-25T18:07:11.068 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:42 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:11.220 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:11.221 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:07:11.222 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:11.222 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.1 M 2024-03-25T18:07:11.222 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 13 M 2024-03-25T18:07:11.222 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:07:11.635 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:41 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:11.784 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 127 k 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 456 k 2024-03-25T18:07:11.785 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:07:12.780 INFO:teuthology.orchestra.run.smithi143.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 127 kB/s | 127 kB 00:00 2024-03-25T18:07:12.780 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:12.780 INFO:teuthology.orchestra.run.smithi143.stdout:Total 127 kB/s | 127 kB 00:00 2024-03-25T18:07:12.781 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:07:12.787 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:07:12.787 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:07:12.845 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:07:12.845 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:07:12.992 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:07:13.112 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:13.567 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:13.842 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:07:13.842 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:13.843 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:13.843 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:14.091 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs2 2024-03-25T18:07:14.663 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:44 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:14.764 INFO:teuthology.orchestra.run.smithi143.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:14.817 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:14.818 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:07:14.818 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:14.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs-devel 2024-03-25T18:07:15.460 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:45 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:15.609 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:15.610 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:15.610 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:07:15.610 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:15.610 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:07:15.610 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:07:15.610 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:15.611 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:07:15.611 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:15.611 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:07:15.611 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:15.611 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 31 k 2024-03-25T18:07:15.611 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 140 k 2024-03-25T18:07:15.611 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:07:16.143 INFO:teuthology.orchestra.run.smithi143.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 59 kB/s | 31 kB 00:00 2024-03-25T18:07:16.143 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:16.143 INFO:teuthology.orchestra.run.smithi143.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T18:07:16.143 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:07:16.146 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:07:16.146 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:07:16.174 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:07:16.174 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:07:16.302 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:07:16.398 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:16.779 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:17.041 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:17.041 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:17.041 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:07:17.041 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:17.041 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:17.041 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:17.200 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados2 2024-03-25T18:07:17.768 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:47 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:17.868 INFO:teuthology.orchestra.run.smithi143.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:17.922 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:17.924 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:07:17.924 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:17.979 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librbd1 2024-03-25T18:07:18.565 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:48 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:18.666 INFO:teuthology.orchestra.run.smithi143.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:18.720 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:18.721 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:07:18.721 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:18.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rados 2024-03-25T18:07:19.343 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:49 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:19.440 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:19.491 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:19.492 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:07:19.492 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:19.544 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rgw 2024-03-25T18:07:20.107 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:50 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:20.204 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:20.256 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:20.257 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:07:20.257 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:20.311 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-cephfs 2024-03-25T18:07:20.873 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:50 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:20.970 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:21.021 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:21.022 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:07:21.022 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:21.077 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rbd 2024-03-25T18:07:21.431 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 310 kB/s | 3.1 MB 00:10 2024-03-25T18:07:21.432 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:21.432 INFO:teuthology.orchestra.run.smithi203.stdout:Total 310 kB/s | 3.1 MB 00:10 2024-03-25T18:07:21.433 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:07:21.446 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:07:21.446 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:07:21.522 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:07:21.522 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:07:21.635 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:51 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:21.733 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:07:21.784 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:21.785 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T18:07:21.785 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:21.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-fuse 2024-03-25T18:07:21.958 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:07:22.012 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:22.036 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:22.036 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:07:22.037 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:07:22.037 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:07:22.037 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:07:22.037 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:22.395 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:52 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:22.542 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:22.543 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:22.543 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-03-25T18:07:22.543 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:22.543 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:07:22.543 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:07:22.544 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:22.544 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:07:22.544 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:22.544 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:07:22.544 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:22.544 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 85 k 2024-03-25T18:07:22.544 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 231 k 2024-03-25T18:07:22.544 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:07:23.148 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:23.149 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:23.149 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:07:23.149 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:23.149 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:23.149 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:23.154 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 140 kB/s | 85 kB 00:00 2024-03-25T18:07:23.154 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:23.155 INFO:teuthology.orchestra.run.smithi143.stdout:Total 139 kB/s | 85 kB 00:00 2024-03-25T18:07:23.155 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:07:23.162 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:07:23.162 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:07:23.218 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:07:23.219 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:07:23.357 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:07:23.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-03-25T18:07:23.468 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:23.871 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:23.979 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:54 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:24.131 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:24.131 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:07:24.132 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:24.133 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 171 k 2024-03-25T18:07:24.133 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 494 k 2024-03-25T18:07:24.133 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:07:24.164 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:24.164 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:24.164 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:07:24.164 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:24.164 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:24.164 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:24.378 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-mirror 2024-03-25T18:07:24.947 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:06:54 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:25.102 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:25.102 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 3.1 M 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 13 M 2024-03-25T18:07:25.103 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:07:25.874 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 98 kB/s | 171 kB 00:01 2024-03-25T18:07:25.875 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:25.875 INFO:teuthology.orchestra.run.smithi203.stdout:Total 98 kB/s | 171 kB 00:01 2024-03-25T18:07:25.876 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:07:25.885 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:07:25.885 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:07:25.946 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:07:25.946 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:07:26.101 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:07:26.215 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:26.639 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:26.913 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:26.914 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:26.914 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:07:26.914 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:26.914 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:26.914 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:07:27.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:07:27.715 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:06:58 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:07:27.865 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:07:27.866 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:27.866 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:07:27.866 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:27.866 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:07:27.866 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:07:27.866 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-25T18:07:27.866 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:07:27.867 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:27.867 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:07:27.867 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:07:27.867 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-03-25T18:07:27.867 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:07:27.867 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 91 M 2024-03-25T18:07:27.867 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 498 M 2024-03-25T18:07:27.868 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:07:31.482 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 496 kB/s | 3.1 MB 00:06 2024-03-25T18:07:31.482 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:31.483 INFO:teuthology.orchestra.run.smithi143.stdout:Total 496 kB/s | 3.1 MB 00:06 2024-03-25T18:07:31.483 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:07:31.493 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:07:31.493 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:07:31.569 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:07:31.569 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:07:31.991 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:07:32.049 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:32.073 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:32.073 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:07:32.073 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:07:32.073 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:07:32.073 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:07:32.073 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:33.237 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:33.238 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:33.238 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:07:33.238 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:33.238 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:33.238 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:33.477 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-nbd 2024-03-25T18:07:34.054 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:07:04 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:34.202 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:34.203 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:34.203 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-03-25T18:07:34.203 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:34.203 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:07:34.203 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:07:34.203 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:34.203 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:07:34.204 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:34.204 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:07:34.204 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:34.204 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 171 k 2024-03-25T18:07:34.204 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 494 k 2024-03-25T18:07:34.204 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:07:35.227 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 168 kB/s | 171 kB 00:01 2024-03-25T18:07:35.228 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:35.228 INFO:teuthology.orchestra.run.smithi143.stdout:Total 167 kB/s | 171 kB 00:01 2024-03-25T18:07:35.228 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:07:35.237 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:07:35.237 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:07:35.296 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:07:35.297 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:07:35.446 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:07:35.551 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:35.990 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:36.278 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:36.278 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:36.278 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:07:36.278 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:36.278 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:36.278 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:07:36.510 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:07:37.083 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:07:07 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:07:37.237 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:07:37.238 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:07:37.239 INFO:teuthology.orchestra.run.smithi143.stdout:Install 2 Packages 2024-03-25T18:07:37.239 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:07:37.239 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 91 M 2024-03-25T18:07:37.239 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 498 M 2024-03-25T18:07:37.239 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:08:37.637 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 329 kB/s | 22 MB 01:09 2024-03-25T18:08:37.832 INFO:teuthology.orchestra.run.smithi143.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 379 kB/s | 22 MB 01:00 2024-03-25T18:10:15.703 INFO:teuthology.orchestra.run.smithi143.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 440 kB/s | 68 MB 02:38 2024-03-25T18:10:15.704 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:15.705 INFO:teuthology.orchestra.run.smithi143.stdout:Total 585 kB/s | 91 MB 02:38 2024-03-25T18:10:15.705 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:10:15.710 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:10:15.710 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:10:16.292 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:10:16.292 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:10:19.384 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:10:26.463 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:10:26.536 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:10:30.053 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:10:30.054 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:10:30.327 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:10:30.327 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:10:30.327 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:10:30.327 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:30.327 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:30.327 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:10:30.327 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:10:30.529 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:10:31.093 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:10:01 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:10:31.241 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:10:31.242 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:10:31.242 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:10:31.242 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:10:31.242 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:10:31.242 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:10:31.242 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:10:31.242 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:10:31.243 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:10:31.243 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:10:31.243 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:10:31.243 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 416 M 2024-03-25T18:10:31.243 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 1.7 G 2024-03-25T18:10:31.243 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:10:41.114 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 361 kB/s | 68 MB 03:13 2024-03-25T18:10:41.115 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:41.115 INFO:teuthology.orchestra.run.smithi203.stdout:Total 480 kB/s | 91 MB 03:13 2024-03-25T18:10:41.116 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:10:41.120 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:10:41.120 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:10:41.713 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:10:41.713 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:10:44.816 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:10:52.014 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:10:52.081 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:10:55.569 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:10:55.569 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:10:55.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:10:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:10:55.839 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:10:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:55.839 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:10:55.839 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:10:56.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:10:56.654 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:10:27 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:10:56.801 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:10:56.802 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:10:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:10:56.802 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:10:56.802 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:10:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:10:56.802 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:10:56.802 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:10:56.802 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:10:56.803 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:10:56.803 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:10:56.803 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 416 M 2024-03-25T18:10:56.803 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.7 G 2024-03-25T18:10:56.803 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:23:13.918 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 578 kB/s | 416 MB 12:17 2024-03-25T18:23:13.919 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:13.919 INFO:teuthology.orchestra.run.smithi203.stdout:Total 578 kB/s | 416 MB 12:17 2024-03-25T18:23:13.919 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:23:13.922 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:23:13.922 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:23:16.383 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:23:16.383 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:23:58.904 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:23:59.021 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:24:06.530 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:24:06.783 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:24:06.784 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:24:06.784 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:24:06.784 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:06.784 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:24:06.784 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:24:07.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:24:07.651 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:23:38 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:24:07.810 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:24:07.810 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:24:07.811 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:24:07.812 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.4 M 2024-03-25T18:24:07.812 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 6.4 M 2024-03-25T18:24:07.812 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:24:10.416 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 561 kB/s | 1.4 MB 00:02 2024-03-25T18:24:10.417 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:24:10.418 INFO:teuthology.orchestra.run.smithi203.stdout:Total 561 kB/s | 1.4 MB 00:02 2024-03-25T18:24:10.418 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:24:10.422 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:24:10.422 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:24:10.467 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:24:10.467 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:24:10.771 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:24:10.863 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:24:11.251 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:24:11.510 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:24:11.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:24:11.511 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:24:11.511 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:11.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:24:11.511 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:24:11.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:24:12.294 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:23:43 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:24:12.442 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:24:12.442 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:24:12.443 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:24:12.444 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 176 M 2024-03-25T18:24:12.444 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 753 M 2024-03-25T18:24:12.444 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:25:25.331 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 477 kB/s | 416 MB 14:54 2024-03-25T18:25:25.331 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:25.331 INFO:teuthology.orchestra.run.smithi143.stdout:Total 477 kB/s | 416 MB 14:54 2024-03-25T18:25:25.331 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:25:25.334 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:25:25.334 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:25:27.797 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:25:27.797 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:26:10.003 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:26:10.099 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:17.257 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:17.520 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:26:17.520 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:26:17.520 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:26:17.521 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:17.521 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:26:17.521 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:26:17.774 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:26:18.338 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:25:48 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================== 2024-03-25T18:26:18.488 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:26:18.489 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:26:18.489 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 1.4 M 2024-03-25T18:26:18.489 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 6.4 M 2024-03-25T18:26:18.489 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:26:20.603 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 691 kB/s | 1.4 MB 00:02 2024-03-25T18:26:20.604 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:26:20.604 INFO:teuthology.orchestra.run.smithi143.stdout:Total 690 kB/s | 1.4 MB 00:02 2024-03-25T18:26:20.604 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:26:20.608 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:26:20.609 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:26:20.653 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:26:20.654 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:26:20.910 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:26:21.010 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:21.419 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:21.700 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:26:21.700 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:26:21.701 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:26:21.701 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:26:21.701 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:26:21.701 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:26:21.871 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:26:22.433 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:25:52 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:26:22.581 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:26:22.582 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 176 M 2024-03-25T18:26:22.583 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 753 M 2024-03-25T18:26:22.583 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 633 kB/s | 176 MB 04:44 2024-03-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi203.stdout:Total 633 kB/s | 176 MB 04:44 2024-03-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:28:56.777 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:28:56.777 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:28:57.833 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:28:57.833 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:29:16.099 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:29:16.195 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:21.113 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:21.376 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:29:21.376 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:29:21.376 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:29:21.376 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:29:21.376 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:29:21.376 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:29:21.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:29:22.182 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:28:53 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:29:22.331 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:29:22.332 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:29:22.332 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:29:22.332 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:29:22.332 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:29:22.332 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:29:22.332 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:29:22.332 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:29:22.333 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:29:22.333 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:29:22.333 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:29:22.333 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 672 M 2024-03-25T18:29:22.333 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 3.1 G 2024-03-25T18:29:22.333 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:30:13.658 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 778 kB/s | 176 MB 03:51 2024-03-25T18:30:13.658 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:13.659 INFO:teuthology.orchestra.run.smithi143.stdout:Total 778 kB/s | 176 MB 03:51 2024-03-25T18:30:13.659 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T18:30:13.662 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T18:30:13.662 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T18:30:14.721 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T18:30:14.721 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T18:30:32.943 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T18:30:33.111 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:30:37.874 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:30:38.142 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:30:38.142 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:30:38.142 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T18:30:38.142 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:38.142 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:30:38.142 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T18:30:38.334 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:30:38.897 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:30:08 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T18:30:39.046 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T18:30:39.047 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:30:39.047 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T18:30:39.047 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:30:39.047 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T18:30:39.047 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:30:39.047 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:30:39.048 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T18:30:39.048 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T18:30:39.048 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T18:30:39.048 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T18:30:39.048 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 672 M 2024-03-25T18:30:39.048 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 3.1 G 2024-03-25T18:30:39.048 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T18:56:08.467 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 429 kB/s | 672 MB 26:46 2024-03-25T18:56:08.468 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:08.468 INFO:teuthology.orchestra.run.smithi203.stdout:Total 429 kB/s | 672 MB 26:46 2024-03-25T18:56:08.468 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:56:08.473 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:56:08.473 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:56:12.443 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:56:12.443 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:57:23.643 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:57:23.745 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:57:33.404 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:57:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:57:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:57:33.657 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:57:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:57:33.657 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:57:33.657 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:57:33.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:57:34.549 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:57:05 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:57:34.646 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:57:34.697 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:57:34.698 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-25T18:57:34.698 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:57:34.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:57:35.362 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:57:06 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:57:35.510 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:57:35.511 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:57:35.512 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:57:35.512 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 20 M 2024-03-25T18:57:35.512 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 103 M 2024-03-25T18:57:35.512 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:58:23.017 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 425 kB/s | 20 MB 00:47 2024-03-25T18:58:23.018 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:58:23.018 INFO:teuthology.orchestra.run.smithi203.stdout:Total 425 kB/s | 20 MB 00:47 2024-03-25T18:58:23.018 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:58:23.022 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:58:23.022 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:58:23.172 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:58:23.172 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T18:58:25.399 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T18:58:25.556 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:26.626 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:26.872 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:58:26.873 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:58:26.873 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T18:58:26.873 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:58:26.873 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:58:26.873 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T18:58:27.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:58:27.637 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:57:58 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T18:58:27.785 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T18:58:27.786 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:58:27.786 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T18:58:27.786 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:58:27.786 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T18:58:27.786 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:58:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:58:27.787 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T18:58:27.787 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T18:58:27.787 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T18:58:27.787 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T18:58:27.787 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 48 M 2024-03-25T18:58:27.787 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 173 M 2024-03-25T18:58:27.787 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T18:59:59.024 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 536 kB/s | 48 MB 01:31 2024-03-25T18:59:59.025 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:59.025 INFO:teuthology.orchestra.run.smithi203.stdout:Total 536 kB/s | 48 MB 01:31 2024-03-25T18:59:59.025 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T18:59:59.028 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T18:59:59.028 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T18:59:59.340 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T18:59:59.340 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:00:04.072 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:00:04.172 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:05.588 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:05.849 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:00:05.849 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:00:05.849 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:00:05.849 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:05.849 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:00:05.849 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:00:06.068 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:00:06.636 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:59:37 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:00:06.784 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:00:06.785 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:00:06.785 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:00:06.785 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:00:06.785 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:00:06.785 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:00:06.785 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:00:06.786 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:00:06.786 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:00:06.786 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:00:06.786 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:00:06.786 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 99 M 2024-03-25T19:00:06.786 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 449 M 2024-03-25T19:00:06.786 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:04:22.958 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 340 kB/s | 672 MB 33:43 2024-03-25T19:04:22.958 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:22.958 INFO:teuthology.orchestra.run.smithi143.stdout:Total 340 kB/s | 672 MB 33:43 2024-03-25T19:04:22.959 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:04:22.963 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:04:22.963 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:04:26.920 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:04:26.920 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:05:10.034 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 333 kB/s | 99 MB 05:03 2024-03-25T19:05:10.035 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:05:10.035 INFO:teuthology.orchestra.run.smithi203.stdout:Total 333 kB/s | 99 MB 05:03 2024-03-25T19:05:10.035 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:05:10.038 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:05:10.038 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:05:10.645 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:05:10.645 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:05:20.902 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:05:21.014 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:24.143 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:24.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:05:24.398 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:05:24.398 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:05:24.398 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:24.398 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:05:24.398 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:05:24.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:05:25.174 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:04:56 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:05:25.323 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:05:25.324 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:05:25.324 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:05:25.324 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:05:25.324 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:05:25.324 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:05:25.325 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:05:25.325 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:05:25.325 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:05:25.325 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:05:25.325 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:05:25.325 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 287 M 2024-03-25T19:05:25.325 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.2 G 2024-03-25T19:05:25.325 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:05:37.538 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:05:38.111 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:05:46.394 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:05:46.668 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:05:46.668 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:05:46.668 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:05:46.668 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:46.668 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:05:46.668 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:05:46.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:05:47.525 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:05:17 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:05:47.621 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:05:47.673 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:05:47.674 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-03-25T19:05:47.674 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:05:47.735 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:05:48.294 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:05:18 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:05:48.440 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:05:48.441 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 20 M 2024-03-25T19:05:48.442 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 103 M 2024-03-25T19:05:48.442 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:06:41.680 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 379 kB/s | 20 MB 00:53 2024-03-25T19:06:41.681 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:06:41.681 INFO:teuthology.orchestra.run.smithi143.stdout:Total 379 kB/s | 20 MB 00:53 2024-03-25T19:06:41.681 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:06:41.685 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:06:41.685 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:06:41.834 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:06:41.834 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:06:44.055 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:06:44.236 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:45.214 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:45.498 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:06:45.499 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:06:45.499 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:06:45.499 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:06:45.499 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:06:45.499 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:06:45.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:06:46.218 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:06:16 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:06:46.367 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 48 M 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 173 M 2024-03-25T19:06:46.369 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:09:09.947 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 340 kB/s | 48 MB 02:23 2024-03-25T19:09:09.947 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:09:09.947 INFO:teuthology.orchestra.run.smithi143.stdout:Total 340 kB/s | 48 MB 02:23 2024-03-25T19:09:09.947 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:09:09.950 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:09:09.950 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:09:10.262 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:09:10.263 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:09:15.010 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:09:15.144 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:09:16.502 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:09:16.756 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:09:16.756 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:09:16.756 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:09:16.756 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:09:16.756 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:09:16.756 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:09:16.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:09:17.472 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:08:47 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:09:17.620 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 99 M 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 449 M 2024-03-25T19:09:17.621 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:13:29.621 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 400 kB/s | 99 MB 04:11 2024-03-25T19:13:29.622 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:29.622 INFO:teuthology.orchestra.run.smithi143.stdout:Total 400 kB/s | 99 MB 04:12 2024-03-25T19:13:29.622 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:13:29.625 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:13:29.625 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:13:30.235 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:13:30.235 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:13:40.495 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:13:40.584 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:13:43.524 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:13:43.798 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:13:43.798 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:13:43.798 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:13:43.798 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:43.798 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:13:43.798 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:13:43.985 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:13:44.551 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:13:14 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:13:44.701 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:13:44.701 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:13:44.701 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:13:44.701 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:13:44.701 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:13:44.701 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:13:44.701 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:13:44.702 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:13:44.702 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:13:44.702 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:13:44.702 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:13:44.702 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 287 M 2024-03-25T19:13:44.702 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 1.2 G 2024-03-25T19:13:44.702 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:17:23.301 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 409 kB/s | 287 MB 11:57 2024-03-25T19:17:23.301 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:23.301 INFO:teuthology.orchestra.run.smithi203.stdout:Total 409 kB/s | 287 MB 11:57 2024-03-25T19:17:23.301 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:17:23.305 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:17:23.305 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:17:25.006 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:17:25.006 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:17:54.454 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:17:54.544 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:02.408 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:02.691 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:18:02.691 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:02.691 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:18:02.691 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:02.691 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:02.691 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:18:02.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:18:03.516 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:17:34 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:18:03.664 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 5.9 M 2024-03-25T19:18:03.665 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 28 M 2024-03-25T19:18:03.666 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:18:12.956 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 655 kB/s | 5.9 MB 00:09 2024-03-25T19:18:12.957 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:12.957 INFO:teuthology.orchestra.run.smithi203.stdout:Total 655 kB/s | 5.9 MB 00:09 2024-03-25T19:18:12.957 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:18:12.962 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:18:12.962 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:18:13.033 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:18:13.033 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:18:13.773 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:18:13.879 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:18:14.483 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:18:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:18:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:14.772 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:18:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:14.772 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:14.773 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:18:14.986 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:18:15.550 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:17:46 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:18:15.700 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:18:15.701 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:15.702 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 109 k 2024-03-25T19:18:15.702 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 440 k 2024-03-25T19:18:15.702 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:18:16.467 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 142 kB/s | 109 kB 00:00 2024-03-25T19:18:16.468 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:16.468 INFO:teuthology.orchestra.run.smithi203.stdout:Total 142 kB/s | 109 kB 00:00 2024-03-25T19:18:16.468 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:18:16.471 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:18:16.472 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:18:16.508 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:18:16.508 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:18:16.624 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:18:16.723 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:18:17.107 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:18:17.371 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:18:17.371 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:17.371 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:18:17.371 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:17.371 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:17.371 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:18:17.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:18:18.156 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:17:49 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:18.307 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:18:18.308 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:18.308 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 5.4 M 2024-03-25T19:18:18.308 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 25 M 2024-03-25T19:18:18.308 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:18:30.493 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 450 kB/s | 5.4 MB 00:12 2024-03-25T19:18:30.494 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:18:30.495 INFO:teuthology.orchestra.run.smithi203.stdout:Total 450 kB/s | 5.4 MB 00:12 2024-03-25T19:18:30.495 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:18:30.499 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:18:30.499 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:18:30.569 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:18:30.570 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:18:31.237 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:18:31.342 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:31.931 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:32.220 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:18:32.220 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:32.220 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:18:32.220 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:18:32.220 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:32.220 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:18:32.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2-debuginfo 2024-03-25T19:18:33.011 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:18:04 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:18:33.162 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 44 M 2024-03-25T19:18:33.163 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 253 M 2024-03-25T19:18:33.164 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:20:15.899 INFO:teuthology.orchestra.run.smithi203.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 441 kB/s | 44 MB 01:42 2024-03-25T19:20:15.900 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:20:15.900 INFO:teuthology.orchestra.run.smithi203.stdout:Total 441 kB/s | 44 MB 01:42 2024-03-25T19:20:15.900 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:20:15.904 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:20:15.904 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:20:16.206 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:20:16.206 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:20:21.271 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:20:21.372 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:23.321 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:23.601 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:20:23.601 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:20:23.602 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:20:23.602 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:20:23.602 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:20:23.602 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:20:23.822 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:20:24.390 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:19:55 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:20:24.539 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 52 M 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 308 M 2024-03-25T19:20:24.540 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:22:38.784 INFO:teuthology.orchestra.run.smithi203.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 395 kB/s | 52 MB 02:14 2024-03-25T19:22:38.784 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:38.784 INFO:teuthology.orchestra.run.smithi203.stdout:Total 395 kB/s | 52 MB 02:14 2024-03-25T19:22:38.785 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:22:38.788 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:22:38.788 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:22:39.134 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:22:39.134 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:22:45.137 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:22:45.323 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:47.615 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:47.875 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:22:47.875 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:47.875 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:22:47.875 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:47.876 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:47.876 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:22:48.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:22:48.658 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:22:19 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:48.811 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:22:48.812 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:48.812 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 615 k 2024-03-25T19:22:48.812 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.1 M 2024-03-25T19:22:48.812 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:22:50.444 INFO:teuthology.orchestra.run.smithi203.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 377 kB/s | 615 kB 00:01 2024-03-25T19:22:50.444 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:50.445 INFO:teuthology.orchestra.run.smithi203.stdout:Total 376 kB/s | 615 kB 00:01 2024-03-25T19:22:50.445 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:22:50.449 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:22:50.449 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:22:50.490 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:22:50.490 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:22:50.655 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:22:50.776 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:22:51.429 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:51.429 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:22:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:51.430 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:51.430 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:22:51.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:22:52.227 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:22:23 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:22:52.379 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 771 k 2024-03-25T19:22:52.381 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.7 M 2024-03-25T19:22:52.381 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:22:55.454 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 251 kB/s | 771 kB 00:03 2024-03-25T19:22:55.455 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:55.456 INFO:teuthology.orchestra.run.smithi203.stdout:Total 251 kB/s | 771 kB 00:03 2024-03-25T19:22:55.456 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:22:55.460 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:22:55.460 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:22:55.503 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:22:55.503 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:22:55.686 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:22:55.788 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:56.188 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:56.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:22:56.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:56.452 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:22:56.452 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:56.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:56.452 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:22:56.678 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:22:57.247 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:22:28 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:22:57.396 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:22:57.396 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:57.396 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:22:57.397 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.1 M 2024-03-25T19:22:57.398 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 4.3 M 2024-03-25T19:22:57.398 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:23:01.557 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 277 kB/s | 1.1 MB 00:04 2024-03-25T19:23:01.564 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:01.564 INFO:teuthology.orchestra.run.smithi203.stdout:Total 277 kB/s | 1.1 MB 00:04 2024-03-25T19:23:01.565 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:23:01.565 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:23:01.565 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:23:01.607 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:23:01.607 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:23:01.825 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:23:01.923 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:23:02.328 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:23:02.583 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:23:02.584 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:02.584 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:23:02.584 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:02.584 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:02.584 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:23:02.817 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:23:03.379 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:22:34 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:23:03.528 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:23:03.529 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:03.529 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:23:03.529 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 337 k 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.0 M 2024-03-25T19:23:03.530 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:23:04.619 INFO:teuthology.orchestra.run.smithi203.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 310 kB/s | 337 kB 00:01 2024-03-25T19:23:04.619 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:04.620 INFO:teuthology.orchestra.run.smithi203.stdout:Total 309 kB/s | 337 kB 00:01 2024-03-25T19:23:04.620 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:23:04.624 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:23:04.624 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:23:04.666 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:23:04.666 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:23:04.812 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:23:04.910 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:23:05.297 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:23:05.558 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:23:05.558 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:05.558 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:23:05.558 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:05.558 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:05.559 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:23:05.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:23:06.336 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:22:37 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:23:06.494 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:23:06.494 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:06.494 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:23:06.494 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:06.495 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:23:06.495 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:23:06.495 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:06.495 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:23:06.495 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:06.495 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:23:06.495 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:06.495 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 509 k 2024-03-25T19:23:06.496 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.5 M 2024-03-25T19:23:06.496 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:23:08.639 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 238 kB/s | 509 kB 00:02 2024-03-25T19:23:08.639 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:23:08.640 INFO:teuthology.orchestra.run.smithi203.stdout:Total 237 kB/s | 509 kB 00:02 2024-03-25T19:23:08.640 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:23:08.644 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:23:08.644 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:23:08.685 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:23:08.686 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:23:08.848 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:23:08.945 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:09.330 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:09.581 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:23:09.582 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:09.582 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:23:09.582 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:23:09.582 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:09.582 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:23:09.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:23:10.363 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:22:41 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:23:10.512 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:23:10.513 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:23:10.514 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:23:10.514 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 52 M 2024-03-25T19:23:10.514 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 298 M 2024-03-25T19:23:10.514 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:24:03.359 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 475 kB/s | 287 MB 10:18 2024-03-25T19:24:03.360 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:03.360 INFO:teuthology.orchestra.run.smithi143.stdout:Total 475 kB/s | 287 MB 10:18 2024-03-25T19:24:03.360 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:24:03.363 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:24:03.363 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:24:05.065 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:24:05.066 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:24:34.423 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:24:34.537 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:42.116 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:42.407 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:24:42.407 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:42.407 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:24:42.407 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:42.407 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:42.407 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:24:42.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:24:43.166 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:24:13 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:24:43.314 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:24:43.315 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:43.316 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 5.9 M 2024-03-25T19:24:43.316 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 28 M 2024-03-25T19:24:43.316 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:24:43.321 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 569 kB/s | 52 MB 01:32 2024-03-25T19:24:43.322 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:43.322 INFO:teuthology.orchestra.run.smithi203.stdout:Total 569 kB/s | 52 MB 01:32 2024-03-25T19:24:43.322 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:24:43.326 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:24:43.326 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:24:43.664 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:24:43.664 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:24:49.613 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:24:49.753 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:51.851 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:52.133 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:24:52.133 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:24:52.133 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:24:52.134 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:52.134 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:24:52.134 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:24:52.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:24:52.955 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 1:24:23 ago on Mon 25 Mar 2024 06:00:29 PM UTC. 2024-03-25T19:24:53.105 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-25T19:24:53.105 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:24:53.105 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-25T19:24:53.105 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:24:53.105 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-25T19:24:53.105 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:24:53.105 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:24:53.105 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-25T19:24:53.106 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-25T19:24:53.106 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-25T19:24:53.106 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:24:53.106 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.3 M 2024-03-25T19:24:53.106 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 5.5 M 2024-03-25T19:24:53.106 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-25T19:24:53.509 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 597 kB/s | 5.9 MB 00:10 2024-03-25T19:24:53.510 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:53.510 INFO:teuthology.orchestra.run.smithi143.stdout:Total 597 kB/s | 5.9 MB 00:10 2024-03-25T19:24:53.511 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:24:53.514 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:24:53.515 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:24:53.586 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:24:53.586 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:24:54.313 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:24:54.423 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:54.975 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:55.252 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:24:55.252 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:55.253 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:24:55.253 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:55.253 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:55.253 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:24:55.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:24:55.987 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:24:25 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:24:56.140 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:24:56.141 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:56.142 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 109 k 2024-03-25T19:24:56.142 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 440 k 2024-03-25T19:24:56.142 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:24:56.414 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 388 kB/s | 1.3 MB 00:03 2024-03-25T19:24:56.415 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:56.415 INFO:teuthology.orchestra.run.smithi203.stdout:Total 387 kB/s | 1.3 MB 00:03 2024-03-25T19:24:56.415 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-25T19:24:56.419 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-25T19:24:56.420 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-25T19:24:56.466 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-25T19:24:56.466 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-25T19:24:56.728 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-25T19:24:56.828 INFO:teuthology.orchestra.run.smithi143.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 158 kB/s | 109 kB 00:00 2024-03-25T19:24:56.828 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:56.829 INFO:teuthology.orchestra.run.smithi143.stdout:Total 158 kB/s | 109 kB 00:00 2024-03-25T19:24:56.829 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:24:56.832 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:24:56.832 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:24:56.837 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:56.868 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:24:56.868 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:24:56.984 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:24:57.077 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:57.313 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:57.451 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:24:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:24:57.573 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-25T19:24:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:57.573 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:24:57.573 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-25T19:24:57.717 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:24:57.717 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:57.718 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:24:57.718 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:24:57.718 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:57.718 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:24:57.784 DEBUG:teuthology.parallel:result is None 2024-03-25T19:24:57.883 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:24:58.447 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:24:28 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:24:58.595 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:24:58.596 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 5.4 M 2024-03-25T19:24:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 25 M 2024-03-25T19:24:58.597 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:25:03.772 INFO:teuthology.orchestra.run.smithi143.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 5.4 MB 00:05 2024-03-25T19:25:03.773 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:03.773 INFO:teuthology.orchestra.run.smithi143.stdout:Total 1.0 MB/s | 5.4 MB 00:05 2024-03-25T19:25:03.773 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:25:03.777 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:25:03.777 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:25:03.845 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:25:03.846 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:25:04.503 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:25:04.603 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:25:05.158 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:25:05.434 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:25:05.435 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:25:05.435 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:25:05.435 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:05.435 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:25:05.435 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:25:05.604 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados2-debuginfo 2024-03-25T19:25:06.171 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:24:36 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:25:06.321 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:25:06.321 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 44 M 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 253 M 2024-03-25T19:25:06.322 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:25:58.909 INFO:teuthology.orchestra.run.smithi143.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 862 kB/s | 44 MB 00:52 2024-03-25T19:25:58.909 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:58.910 INFO:teuthology.orchestra.run.smithi143.stdout:Total 862 kB/s | 44 MB 00:52 2024-03-25T19:25:58.910 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:25:58.913 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:25:58.913 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:25:59.212 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:25:59.212 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:26:04.261 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:26:04.348 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:06.180 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:06.465 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:06.465 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:26:06.465 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:26:06.465 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:06.465 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:26:06.466 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:26:06.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:26:07.215 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:25:37 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:26:07.362 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:26:07.363 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:26:07.363 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:26:07.363 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:26:07.363 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:26:07.363 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:26:07.363 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:26:07.363 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:26:07.363 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:26:07.364 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:26:07.364 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:26:07.364 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 52 M 2024-03-25T19:26:07.364 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 308 M 2024-03-25T19:26:07.364 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:27:15.660 INFO:teuthology.orchestra.run.smithi143.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 777 kB/s | 52 MB 01:08 2024-03-25T19:27:15.661 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:15.661 INFO:teuthology.orchestra.run.smithi143.stdout:Total 777 kB/s | 52 MB 01:08 2024-03-25T19:27:15.661 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:27:15.664 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:27:15.664 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:27:16.007 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:27:16.008 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:27:21.973 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:27:22.058 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:24.191 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:24.479 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:27:24.480 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:24.480 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:27:24.480 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:24.480 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:24.480 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:27:24.669 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:27:25.229 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:26:55 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:27:25.377 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:27:25.378 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 615 k 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 2.1 M 2024-03-25T19:27:25.379 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:27:26.486 INFO:teuthology.orchestra.run.smithi143.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 555 kB/s | 615 kB 00:01 2024-03-25T19:27:26.487 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:26.488 INFO:teuthology.orchestra.run.smithi143.stdout:Total 555 kB/s | 615 kB 00:01 2024-03-25T19:27:26.488 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:27:26.492 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:27:26.492 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:27:26.533 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:27:26.534 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:27:26.704 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:27:26.827 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:27:27.213 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:27:27.487 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:27:27.487 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:27.487 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:27:27.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:27.487 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:27.487 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:27:27.676 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:27:28.236 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:26:58 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:27:28.388 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:27:28.389 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:28.389 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:27:28.389 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 771 k 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 2.7 M 2024-03-25T19:27:28.390 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:27:30.889 INFO:teuthology.orchestra.run.smithi143.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 308 kB/s | 771 kB 00:02 2024-03-25T19:27:30.890 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:30.890 INFO:teuthology.orchestra.run.smithi143.stdout:Total 308 kB/s | 771 kB 00:02 2024-03-25T19:27:30.890 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:27:30.894 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:27:30.894 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:27:30.937 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:27:30.937 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:27:31.117 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:27:31.229 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:27:31.658 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:27:31.926 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:27:31.926 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:31.926 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:27:31.926 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:31.926 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:31.927 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:27:32.104 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:27:32.663 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:27:02 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:27:32.812 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:27:32.812 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 1.1 M 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 4.3 M 2024-03-25T19:27:32.813 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:27:35.242 INFO:teuthology.orchestra.run.smithi143.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 474 kB/s | 1.1 MB 00:02 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi143.stdout:Total 474 kB/s | 1.1 MB 00:02 2024-03-25T19:27:35.243 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:27:35.247 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:27:35.247 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:27:35.291 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:27:35.291 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:27:35.531 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:27:35.641 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:36.042 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:36.325 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:36.326 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:36.326 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:27:36.326 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:36.326 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:36.326 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:27:36.531 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:27:37.094 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:27:07 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:27:37.245 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:27:37.246 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:37.246 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:27:37.246 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:37.246 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:27:37.246 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:27:37.246 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:37.246 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:27:37.247 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:37.247 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:27:37.247 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:37.247 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 337 k 2024-03-25T19:27:37.247 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 1.0 M 2024-03-25T19:27:37.247 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:27:39.303 INFO:teuthology.orchestra.run.smithi143.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 164 kB/s | 337 kB 00:02 2024-03-25T19:27:39.304 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:39.305 INFO:teuthology.orchestra.run.smithi143.stdout:Total 164 kB/s | 337 kB 00:02 2024-03-25T19:27:39.305 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:27:39.309 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:27:39.309 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:27:39.349 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:27:39.349 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:27:39.493 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:27:39.594 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:39.979 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:40.258 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:27:40.258 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:40.258 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:27:40.258 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:40.258 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:40.258 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:27:40.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:27:41.023 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:27:11 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:27:41.171 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:27:41.172 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:41.172 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:27:41.172 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:41.172 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:27:41.172 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:27:41.172 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:41.173 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:27:41.173 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:41.173 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:27:41.173 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:41.173 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 509 k 2024-03-25T19:27:41.173 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 2.5 M 2024-03-25T19:27:41.173 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:27:43.131 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 260 kB/s | 509 kB 00:01 2024-03-25T19:27:43.131 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:43.132 INFO:teuthology.orchestra.run.smithi143.stdout:Total 260 kB/s | 509 kB 00:01 2024-03-25T19:27:43.132 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:27:43.136 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:27:43.136 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:27:43.176 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:27:43.176 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:27:43.337 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:27:43.438 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:43.812 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:44.090 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:44.090 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:44.090 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:27:44.090 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:44.091 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:44.091 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:27:44.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:27:44.853 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:27:14 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:27:45.001 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:27:45.002 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:27:45.003 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:27:45.003 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 52 M 2024-03-25T19:27:45.003 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 298 M 2024-03-25T19:27:45.003 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:28:44.486 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 887 kB/s | 52 MB 00:59 2024-03-25T19:28:44.487 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:44.487 INFO:teuthology.orchestra.run.smithi143.stdout:Total 887 kB/s | 52 MB 00:59 2024-03-25T19:28:44.487 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:28:44.489 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:28:44.489 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:28:44.824 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:28:44.824 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:28:50.724 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:28:50.916 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:28:53.009 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:28:53.298 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:28:53.298 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:28:53.298 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:28:53.298 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:53.299 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:28:53.299 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:28:53.512 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:28:54.075 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 1:28:24 ago on Mon 25 Mar 2024 06:00:30 PM UTC. 2024-03-25T19:28:54.223 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 1.3 M 2024-03-25T19:28:54.224 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 5.5 M 2024-03-25T19:28:54.225 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-25T19:28:56.103 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 683 kB/s | 1.3 MB 00:01 2024-03-25T19:28:56.103 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:56.103 INFO:teuthology.orchestra.run.smithi143.stdout:Total 682 kB/s | 1.3 MB 00:01 2024-03-25T19:28:56.103 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-25T19:28:56.107 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-25T19:28:56.152 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-25T19:28:56.153 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-25T19:28:56.396 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-25T19:28:56.509 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:56.936 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:57.220 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:28:57.220 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:28:57.220 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-25T19:28:57.221 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:57.221 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:28:57.221 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-25T19:28:57.410 DEBUG:teuthology.parallel:result is None 2024-03-25T19:28:57.410 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:28:57.557 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:28:57.577 INFO:teuthology.orchestra.run.smithi143.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:28:57.578 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:28:57.578 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:28:57.579 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:28:57.717 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:28:57.742 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:28:57.742 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:28:57.742 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:28:57.744 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:28:57.744 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:28:57.744 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:28:57.771 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:28:57.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:28:57.813 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:28:57.813 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:28:57.813 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:28:57.840 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:28:57.907 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:28:57.907 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:28:57.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:28:58.010 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:28:58.010 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:28:58.010 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:28:58.037 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:28:58.102 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:28:58.103 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:28:58.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:28:58.201 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:28:58.201 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:28:58.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:28:58.228 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:28:58.295 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:28:58.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:28:58.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:28:58.392 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:28:58.474 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:28:58.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /var/log/ceph 2024-03-25T19:28:58.477 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-03-25T19:28:58.504 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:28:58.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:28:58.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:28:58.571 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:28:58.571 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:28:58.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:28:58.649 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:28:58.649 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:28:58.649 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621309', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi143.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUEFSADnVvILqzvtrWtHAF9jYxUwPIXtnJtROHs2BTLVMNONytR0y9dNqmYCAahSYtKDAkrpvk18RnY3FK4g8Q=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBq7nuCIIYPLhS6F1kW7ValjeH/QJoJLAYq3KIIS6zBJhjXFs5rWzRdx5QRsovxS0X92BplHJpKsJ1Ln2Eo+s/M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:28:58.650 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:28:58.677 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:28:58.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:28:58.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:28:58.760 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:28:58.760 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:28:58.811 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:28:58.811 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-03-25T19:28:58.866 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:28:58.866 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:58.866 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:28:58.866 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:58.866 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:58.867 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-25 18:07:32.318458854 +0000 2024-03-25T19:28:58.867 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-25 17:59:02.452653495 +0000 2024-03-25T19:28:58.867 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-25 17:59:02.452653495 +0000 2024-03-25T19:28:58.867 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-25 17:59:02.452653495 +0000 2024-03-25T19:28:58.867 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:28:58.931 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-25T19:28:58.932 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-25T19:28:58.932 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000240626 s, 2.1 MB/s 2024-03-25T19:28:58.932 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:28:58.988 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-25 18:07:32.319458829 +0000 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-25 17:59:02.789644994 +0000 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-25 17:59:02.789644994 +0000 2024-03-25T19:28:59.043 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-25 17:59:02.789644994 +0000 2024-03-25T19:28:59.044 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:28:59.108 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-25T19:28:59.108 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-25T19:28:59.109 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000196998 s, 2.6 MB/s 2024-03-25T19:28:59.110 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:28:59.165 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-03-25T19:28:59.220 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:28:59.220 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:59.220 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:28:59.220 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:59.220 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:59.220 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-25 18:07:32.319458829 +0000 2024-03-25T19:28:59.220 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-25 17:59:03.109636922 +0000 2024-03-25T19:28:59.220 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-25 17:59:03.109636922 +0000 2024-03-25T19:28:59.221 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-25 17:59:03.109636922 +0000 2024-03-25T19:28:59.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:28:59.286 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-25T19:28:59.286 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-25T19:28:59.286 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000186006 s, 2.8 MB/s 2024-03-25T19:28:59.287 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:28:59.343 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-03-25T19:28:59.397 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:28:59.398 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:59.398 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:28:59.398 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:59.398 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:59.398 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-25 18:07:32.319458829 +0000 2024-03-25T19:28:59.398 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-25 17:59:03.641623502 +0000 2024-03-25T19:28:59.398 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-25 17:59:03.641623502 +0000 2024-03-25T19:28:59.398 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-25 17:59:03.641623502 +0000 2024-03-25T19:28:59.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:28:59.463 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-25T19:28:59.463 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-25T19:28:59.463 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000191465 s, 2.7 MB/s 2024-03-25T19:28:59.465 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:28:59.520 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:28:59.521 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:28:59.521 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:28:59.538 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:28:59.538 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-25 18:07:22.286951301 +0000 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-25 17:59:19.200764367 +0000 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-25 17:59:19.200764367 +0000 2024-03-25T19:28:59.596 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-25 17:59:19.200764367 +0000 2024-03-25T19:28:59.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:28:59.666 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-25T19:28:59.666 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-25T19:28:59.666 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000281252 s, 1.8 MB/s 2024-03-25T19:28:59.667 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:28:59.724 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-03-25T19:28:59.780 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:28:59.780 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:59.780 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T19:28:59.781 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:59.781 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:59.781 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-25 18:07:22.286951301 +0000 2024-03-25T19:28:59.781 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-25 17:59:19.551755577 +0000 2024-03-25T19:28:59.781 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-25 17:59:19.551755577 +0000 2024-03-25T19:28:59.781 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-25 17:59:19.551755577 +0000 2024-03-25T19:28:59.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:28:59.847 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-25T19:28:59.848 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-25T19:28:59.848 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000213503 s, 2.4 MB/s 2024-03-25T19:28:59.849 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:28:59.906 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-03-25T19:28:59.962 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:28:59.962 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:28:59.962 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:28:59.962 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:28:59.962 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:28:59.962 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-25 18:07:22.286951301 +0000 2024-03-25T19:28:59.963 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-25 17:59:19.888747137 +0000 2024-03-25T19:28:59.963 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-25 17:59:19.888747137 +0000 2024-03-25T19:28:59.963 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-25 17:59:19.888747137 +0000 2024-03-25T19:28:59.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:29:00.028 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-25T19:29:00.029 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-25T19:29:00.029 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000202517 s, 2.5 MB/s 2024-03-25T19:29:00.030 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:29:00.087 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-03-25T19:29:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:29:00.142 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:29:00.142 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T19:29:00.142 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:29:00.142 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:29:00.142 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-25 18:07:22.287951275 +0000 2024-03-25T19:29:00.142 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-25 17:59:20.219738848 +0000 2024-03-25T19:29:00.142 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-25 17:59:20.219738848 +0000 2024-03-25T19:29:00.143 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-25 17:59:20.219738848 +0000 2024-03-25T19:29:00.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:29:00.208 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-25T19:29:00.208 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-25T19:29:00.208 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000208431 s, 2.5 MB/s 2024-03-25T19:29:00.209 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:29:00.265 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:00.265 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:29:00.265 INFO:tasks.ceph:Generating config... 2024-03-25T19:29:00.266 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:29:00.267 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:29:00.267 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:29:00.267 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:29:00.267 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:29:00.267 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:29:00.267 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:29:00.267 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:29:00.267 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:29:00.267 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:29:00.267 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:29:00.267 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:29:00.267 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:29:00.267 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:29:00.267 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:29:00.268 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:29:00.268 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:29:00.268 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:29:00.268 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:29:00.268 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:29:00.268 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:29:00.268 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:29:00.268 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:29:00.268 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:29:00.268 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:29:00.268 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:29:00.268 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:29:00.268 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:29:00.268 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:29:00.268 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:29:00.268 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:29:00.269 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:29:00.270 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:29:00.270 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:29:00.270 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:29:00.270 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:29:00.270 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:29:00.270 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:29:00.270 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:29:00.270 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:29:00.318 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:29:00.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:29:00.365 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:29:00.392 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.143:6789]'), ('mon.c', '[v1:172.21.15.143:6790]'), ('mon.b', '[v1:172.21.15.203:6789]')] 2024-03-25T19:29:00.392 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.143:6789],[v1:172.21.15.143:6790],[v1:172.21.15.203:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:29:00.393 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:00.393 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:29:00.447 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.143:6789]' --addv c '[v1:172.21.15.143:6790]' --addv b '[v1:172.21.15.203:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:00.531 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:00.531 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: generated fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:00.531 INFO:teuthology.orchestra.run.smithi143.stdout:setting min_mon_release = quincy 2024-03-25T19:29:00.531 INFO:teuthology.orchestra.run.smithi143.stdout:epoch 0 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:min_mon_release 17 (quincy) 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:election_strategy: 3 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:00.532 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:29:00.533 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:29:00.547 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d1ef59af-7ebc-4de5-a205-9cc943985e2f... 2024-03-25T19:29:00.548 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:29:00.590 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:29:00.637 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2024-03-25T19:29:00.637 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.638 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2024-03-25T19:29:00.639 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2024-03-25T19:29:00.640 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = "[v1:172.21.15.143:6789],[v1:172.21.15.143:6790],[v1:172.21.15.203:6789]" 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: mon client directed command retry = 5 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: mon election default strategy = 3 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: ms bind msgr2 = False 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject socket failures = 5000 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: ms type = async 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: osd heartbeat grace = 80 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:29:00.641 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug trim objects = true 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2024-03-25T19:29:00.642 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: bdev async discard = True 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: bdev enable discard = True 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore allocator = stupid 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore zero block detection = True 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 1 2024-03-25T19:29:00.643 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 1 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: debug monc = 20 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 10 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: osd class default list = * 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: osd class load list = * 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: osd client watch timeout = 120 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify cached snaps = True 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify missing on start = True 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:29:00.644 INFO:teuthology.orchestra.run.smithi143.stdout: osd fast shutdown = False 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: osd max markdown count = 1000 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock override recovery settings = True 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: osd op queue = debug_random 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: osd op queue cut off = debug_random 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: osd shutdown pgref assert = True 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:29:00.645 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.646 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: # 1m isn't quite enough 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: mon min osdmap epochs = 50 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd crush smoke test = False 2024-03-25T19:29:00.647 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune interval = 2 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune min = 15 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: mon scrub interval = 300 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on pool no app = False 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: paxos service trim min = 10 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: debug monc = 20 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-03-25T19:29:00.648 INFO:teuthology.orchestra.run.smithi143.stdout: debug objecter = 20 2024-03-25T19:29:00.649 INFO:teuthology.orchestra.run.smithi143.stdout: debug rados = 20 2024-03-25T19:29:00.649 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2024-03-25T19:29:00.649 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.c] 2024-03-25T19:29:00.649 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.b] 2024-03-25T19:29:00.649 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-03-25T19:29:00.649 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-03-25T19:29:00.649 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-03-25T19:29:00.650 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:29:00.651 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "[v1:172.21.15.143:6789],[v1:172.21.15.143:6790],[v1:172.21.15.203:6789]" 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: mon client directed command retry = 5 2024-03-25T19:29:00.652 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 3 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: ms bind msgr2 = False 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject socket failures = 5000 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: ms type = async 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: osd heartbeat grace = 80 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-03-25T19:29:00.653 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.654 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: bdev async discard = True 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: bdev enable discard = True 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore allocator = stupid 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 1 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 1 2024-03-25T19:29:00.655 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 10 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: osd class default list = * 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: osd class load list = * 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: osd client watch timeout = 120 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:29:00.656 INFO:teuthology.orchestra.run.smithi203.stdout: osd fast shutdown = False 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: osd max markdown count = 1000 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: osd shutdown pgref assert = True 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:29:00.657 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:29:00.658 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 50 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd crush smoke test = False 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-03-25T19:29:00.659 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on pool no app = False 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 10 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: debug objecter = 20 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: debug rados = 20 2024-03-25T19:29:00.660 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-03-25T19:29:00.661 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-03-25T19:29:00.661 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-03-25T19:29:00.662 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:29:00.662 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:29:00.717 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:29:00.718 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:00.718 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:29:00.732 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:00.732 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:29:00.788 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:29:00.788 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:00.788 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:29:00.788 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:29:00.867 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:00.867 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:00.922 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:29:00.922 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:00.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:29:00.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:29:00.961 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:00.961 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:01.016 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:29:01.016 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:29:01.017 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:29:01.071 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:29:01.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:29:01.136 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:29:01.139 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:29:01.139 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:29:01.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:29:01.181 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:29:01.197 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:29:01.239 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:29:01.254 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:29:01.254 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:29:01.255 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:29:01.281 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:01.281 INFO:tasks.ceph:role: osd.0 2024-03-25T19:29:01.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:29:01.281 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:29:01.355 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:29:01.355 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:29:01.355 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:29:01.355 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:29:01.355 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:29:01.355 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:29:01.356 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:29:01.356 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:29:01.356 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:29:01.356 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:29:01.871 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2024-03-25T19:29:01.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-03-25T19:29:01.873 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:29:02.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:29:02.056 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:29:02.123 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:02.123 INFO:tasks.ceph:role: osd.1 2024-03-25T19:29:02.123 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:29:02.124 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:29:02.197 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:29:02.197 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:29:02.198 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:29:02.198 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:29:02.198 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:29:02.198 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:29:02.198 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:29:02.198 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:29:02.198 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:29:02.198 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:29:02.646 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2024-03-25T19:29:02.647 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-03-25T19:29:02.648 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:29:02.740 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:29:02.773 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:29:02.840 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:02.840 INFO:tasks.ceph:role: osd.2 2024-03-25T19:29:02.840 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:29:02.841 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:29:02.915 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:29:02.915 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:29:02.915 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:29:02.915 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:29:02.915 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:29:02.915 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:29:02.915 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:29:02.916 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:29:02.916 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:29:02.916 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:29:03.403 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2024-03-25T19:29:03.405 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-03-25T19:29:03.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:29:03.537 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:29:03.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:29:03.635 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:03.635 INFO:tasks.ceph:role: osd.3 2024-03-25T19:29:03.635 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:29:03.635 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:29:03.709 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:29:03.709 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:29:03.709 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:29:03.709 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:29:03.710 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:29:03.710 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:29:03.710 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:29:03.710 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:29:03.710 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:29:03.710 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:29:04.224 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2024-03-25T19:29:04.227 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-03-25T19:29:04.227 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:29:04.359 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:29:04.397 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:04.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:04.496+0000 7f2759637640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:29:04.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:04.496+0000 7f2759637640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:29:04.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:04.496+0000 7f2759637640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:29:04.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:04.496+0000 7f2759637640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:29:04.496 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:04.496+0000 7f2759637640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:29:04.497 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:04.497+0000 7f2759637640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:29:07.348 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:29:07.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:07.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:07.470+0000 7f18d7819640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:29:07.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:07.470+0000 7f18d7819640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:29:07.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:07.471+0000 7f18d7819640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:29:07.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:07.471+0000 7f18d7819640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:29:07.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:07.471+0000 7f18d7819640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:29:07.471 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:07.471+0000 7f18d7819640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:29:10.316 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:29:10.348 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:10.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:10.439+0000 7f1ecc6c2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:29:10.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:10.439+0000 7f1ecc6c2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:29:10.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:10.439+0000 7f1ecc6c2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:29:10.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:10.439+0000 7f1ecc6c2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:29:10.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:10.439+0000 7f1ecc6c2640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:29:10.439 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:10.440+0000 7f1ecc6c2640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:29:13.290 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:29:13.321 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:13.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:13.412+0000 7ff38852a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:29:13.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:13.412+0000 7ff38852a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:29:13.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:13.412+0000 7ff38852a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:29:13.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:13.412+0000 7ff38852a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:29:13.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:13.412+0000 7ff38852a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:29:13.413 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:13.413+0000 7ff38852a640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:29:16.257 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:29:16.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:29:16.319 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:16.319 INFO:tasks.ceph:role: osd.4 2024-03-25T19:29:16.319 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:29:16.320 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:29:16.397 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:29:16.918 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-25T19:29:16.920 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-25T19:29:16.920 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:29:17.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:29:17.086 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:29:17.155 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:17.155 INFO:tasks.ceph:role: osd.5 2024-03-25T19:29:17.155 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:29:17.155 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:29:17.231 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:29:17.682 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-25T19:29:17.684 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-25T19:29:17.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:29:17.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:29:17.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:29:17.880 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:17.880 INFO:tasks.ceph:role: osd.6 2024-03-25T19:29:17.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:29:17.881 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:29:17.955 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:29:18.442 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-25T19:29:18.444 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-25T19:29:18.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:29:18.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:29:18.621 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:29:18.690 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:29:18.690 INFO:tasks.ceph:role: osd.7 2024-03-25T19:29:18.690 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:29:18.690 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:29:18.765 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:29:18.765 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:29:18.765 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:29:18.765 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:29:18.765 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:29:18.766 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:29:18.766 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:29:18.766 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:29:18.766 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:29:18.766 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:29:19.281 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-25T19:29:19.283 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-25T19:29:19.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:29:19.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:29:19.467 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:19.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:19.565+0000 7f0bf8287640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:29:19.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:19.566+0000 7f0bf8287640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:29:19.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:19.566+0000 7f0bf8287640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:29:19.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:19.566+0000 7f0bf8287640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:29:19.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:19.566+0000 7f0bf8287640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:29:19.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:19.566+0000 7f0bf8287640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:29:22.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:29:22.439 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:22.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:22.529+0000 7f575f20d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:29:22.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:22.530+0000 7f575f20d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:29:22.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:22.530+0000 7f575f20d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:29:22.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:22.530+0000 7f575f20d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:29:22.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:22.530+0000 7f575f20d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:29:22.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:22.530+0000 7f575f20d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:29:25.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:29:25.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:25.493 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:25.492+0000 7f6d804ce640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:29:25.493 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:25.492+0000 7f6d804ce640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:29:25.493 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:25.492+0000 7f6d804ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:29:25.493 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:25.492+0000 7f6d804ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:29:25.493 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:25.492+0000 7f6d804ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:29:25.494 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:25.493+0000 7f6d804ce640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:29:28.339 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:29:28.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:28.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:28.458+0000 7f007441e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:29:28.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:28.458+0000 7f007441e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:29:28.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:28.458+0000 7f007441e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:29:28.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:28.458+0000 7f007441e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:29:28.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:28.458+0000 7f007441e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:29:28.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:28.458+0000 7f007441e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:29:31.296 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:29:31.326 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:29:31.327 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:31.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:29:31.360 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:31.360 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:29:31.431 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:31.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:29:31.501 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:31.501 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:29:31.570 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:31.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:29:31.637 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:31.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:29:31.669 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:31.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:29:31.739 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:31.739 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:29:31.808 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:31.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:29:31.877 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:31.878 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:29:31.947 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:31.947 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:29:31.963 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:31.963 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:29:32.003 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:29:32.003 DEBUG:teuthology.orchestra.run.smithi143:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:29:32.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:29:32.048 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.y] 2024-03-25T19:29:32.048 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQB90AFmJP9RBBAAIygyy2evSyOL5VPi4D36LQ== 2024-03-25T19:29:32.048 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.0] 2024-03-25T19:29:32.048 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCA0AFmgBamHRAARy3bqmmVm06Z96FTq8/9Mw== 2024-03-25T19:29:32.049 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.1] 2024-03-25T19:29:32.049 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCD0AFmI1oaHBAAC/+LZIRqDnT8YW3QlWD13w== 2024-03-25T19:29:32.049 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.2] 2024-03-25T19:29:32.049 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCG0AFm3L49GhAArte9bdohSqw2GI+9fpVfIQ== 2024-03-25T19:29:32.049 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.3] 2024-03-25T19:29:32.050 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCJ0AFmUhyjGBAACg2l+uRJ27W/10ngcQTB+Q== 2024-03-25T19:29:32.050 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.x] 2024-03-25T19:29:32.050 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQB90AFmZ64+CBAArv0xmutQd22rKOmsaARsCA== 2024-03-25T19:29:32.050 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.4] 2024-03-25T19:29:32.050 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCP0AFmrALJIRAAf/LqtHo0hbFwytUaE6JEmQ== 2024-03-25T19:29:32.050 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.5] 2024-03-25T19:29:32.051 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCS0AFmu96iHxAAkr2iqsCHLVyVF2SEXs/puw== 2024-03-25T19:29:32.051 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.6] 2024-03-25T19:29:32.051 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCV0AFm/dB5HRAAiapaAh/8MXSm/3wKUSxBCw== 2024-03-25T19:29:32.051 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.7] 2024-03-25T19:29:32.051 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQCY0AFmm85qGxAAuqvXTOT1MpXT2RQKZbzDLQ== 2024-03-25T19:29:32.052 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2024-03-25T19:29:32.052 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQB90AFmlvjsChAACbmDSa4nJsD76hS7L1D5Pg== 2024-03-25T19:29:32.052 INFO:teuthology.orchestra.run.smithi143.stdout:[client.1] 2024-03-25T19:29:32.052 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQB90AFmo0BVDhAAbippqzT+mCz1KohAAwrT2g== 2024-03-25T19:29:32.070 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.y] 2024-03-25T19:29:32.070 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQB90AFmJP9RBBAAIygyy2evSyOL5VPi4D36LQ== 2024-03-25T19:29:32.070 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-03-25T19:29:32.070 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCA0AFmgBamHRAARy3bqmmVm06Z96FTq8/9Mw== 2024-03-25T19:29:32.070 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCD0AFmI1oaHBAAC/+LZIRqDnT8YW3QlWD13w== 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCG0AFm3L49GhAArte9bdohSqw2GI+9fpVfIQ== 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCJ0AFmUhyjGBAACg2l+uRJ27W/10ngcQTB+Q== 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQB90AFmZ64+CBAArv0xmutQd22rKOmsaARsCA== 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCP0AFmrALJIRAAf/LqtHo0hbFwytUaE6JEmQ== 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCS0AFmu96iHxAAkr2iqsCHLVyVF2SEXs/puw== 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.6] 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCV0AFm/dB5HRAAiapaAh/8MXSm/3wKUSxBCw== 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.7] 2024-03-25T19:29:32.071 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCY0AFmm85qGxAAuqvXTOT1MpXT2RQKZbzDLQ== 2024-03-25T19:29:32.072 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-03-25T19:29:32.072 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQB90AFmlvjsChAACbmDSa4nJsD76hS7L1D5Pg== 2024-03-25T19:29:32.072 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2024-03-25T19:29:32.072 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQB90AFmo0BVDhAAbippqzT+mCz1KohAAwrT2g== 2024-03-25T19:29:32.073 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:29:32.096 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:29:32.167 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.222 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.224 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.275 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.330 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.384 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:29:32.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:29:32.438 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.440 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.601 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:29:32.655 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:29:32.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:29:32.712 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:29:32.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:29:32.769 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:29:32.769 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:29:32.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:29:32.893 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:29:32.919 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:29:32.986 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:29:33.083 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:29:33.109 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:29:33.137 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:29:33.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:29:33.262 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:33.264 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:29:33.316 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:29:33.317 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:29:33.317 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:29:33.317 DEBUG:teuthology.orchestra.run.smithi143:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:29:33.319 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:29:33.319 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:29:33.320 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:29:33.320 DEBUG:teuthology.orchestra.run.smithi143:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:29:33.321 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:29:33.322 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:29:33.322 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:29:33.322 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:29:33.358 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:29:33.359 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:29:33.359 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:29:33.359 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:29:33.361 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:29:33.361 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:29:33.361 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:29:33.363 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:29:33.363 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:29:33.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:29:33.464 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.464+0000 7fd7c913d640 10 monclient: get_monmap_and_config 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.464+0000 7fd7c913d640 10 monclient: build_initial_monmap 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.464+0000 7fd7c913d640 10 monclient: monmap: 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:33.466324+0000 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:33.466324+0000 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:29:33.465 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:29:33.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 1 Processor -- start 2024-03-25T19:29:33.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 1 -- start start 2024-03-25T19:29:33.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 10 monclient: init 2024-03-25T19:29:33.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:33.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:33.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c40a9920 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:33.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c40b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:33.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c4146450 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:33.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c406b100 con 0x7fd7c40a9920 2024-03-25T19:29:33.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40b2940 con 0x7fd7c40b45a0 2024-03-25T19:29:33.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c413fec0 con 0x7fd7c4146450 2024-03-25T19:29:33.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:33.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c913d640 10 monclient(hunting): authenticate will time out at 5989.810380s 2024-03-25T19:29:33.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7c413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:33.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c359e640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7c414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:33.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.465+0000 7fd7c259c640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7c40b45a0 legacy=0x7fd7c4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:33.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.666+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7c413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:33.666 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.666+0000 7fd7c359e640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7c414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:33.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:33.666+0000 7fd7c259c640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7c40b45a0 legacy=0x7fd7c4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:34.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:34.067+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7c413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:34.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:34.067+0000 7fd7c359e640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7c414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:34.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:34.067+0000 7fd7c259c640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7c40b45a0 legacy=0x7fd7c4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:34.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:34.868+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7c413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:34.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:34.868+0000 7fd7c359e640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7c414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:34.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:34.868+0000 7fd7c259c640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7c40b45a0 legacy=0x7fd7c4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 10 monclient(hunting): tick 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7c414dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7c40b45a0 legacy=0x7fd7c4145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7c413f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b0003680 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c4146450 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c40a9920 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c406b100 con 0x7fd7c40a9920 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40b2940 con 0x7fd7b0003680 2024-03-25T19:29:36.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c413fec0 con 0x7fd7c4146450 2024-03-25T19:29:36.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c259c640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b00093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:36.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c0d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:36.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c359e640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:36.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.465+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:36.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.666+0000 7fd7c259c640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b00093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:36.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.666+0000 7fd7c359e640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:36.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:36.666+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:37.067 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:37.067+0000 7fd7c259c640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b00093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:37.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:37.067+0000 7fd7c359e640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:37.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:37.067+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:37.868 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:37.868+0000 7fd7c259c640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b00093a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:37.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:37.868+0000 7fd7c359e640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:37.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:37.868+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:39.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.466+0000 7fd7c0d99640 10 monclient(hunting): tick 2024-03-25T19:29:39.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.466+0000 7fd7c0d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:29:39.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.466+0000 7fd7c0d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:29:39.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.466+0000 7fd7c0d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:29:39.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.466+0000 7fd7c0d99640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b00093a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:39.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.466+0000 7fd7c0d99640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:39.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:39.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:39.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c40a9920 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:39.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c4146450 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:39.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b0003680 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:39.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0011350 con 0x7fd7c40a9920 2024-03-25T19:29:39.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c406b100 con 0x7fd7b0003680 2024-03-25T19:29:39.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40b2940 con 0x7fd7c4146450 2024-03-25T19:29:39.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c0d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:39.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c259c640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b0010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:39.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:39.470 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.467+0000 7fd7c359e640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:39.667 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.668+0000 7fd7c259c640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b0010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:39.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.668+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:39.668 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:39.668+0000 7fd7c359e640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:40.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:40.069+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:40.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:40.069+0000 7fd7c259c640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b0010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:40.068 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:40.069+0000 7fd7c359e640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:40.394 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:00:06.930 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:40.662 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:00:07.227 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:40.663 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:00:07.227 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:40.703 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:00:07.239 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:40.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:40.870+0000 7fd7c259c640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b0010360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6790/0 2024-03-25T19:29:40.869 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:40.870+0000 7fd7c2d9d640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.143:6789/0 2024-03-25T19:29:40.870 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:40.870+0000 7fd7c359e640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-03-25T19:29:40.983 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:00:07.548 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:40.983 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:00:07.548 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 10 monclient(hunting): tick 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 1 -- >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b0010360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 1 -- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b0003680 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c4146450 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c40a9920 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c413fec0 con 0x7fd7c40a9920 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0011350 con 0x7fd7b0003680 2024-03-25T19:29:42.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c406b100 con 0x7fd7c4146450 2024-03-25T19:29:42.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.467+0000 7fd7c0d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:42.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.523+0000 7fd7c359e640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 0x7fd7b000cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51078/0 (socket says 172.21.15.143:51078) 2024-03-25T19:29:42.523 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.523+0000 7fd7c359e640 1 -- 172.21.15.143:0/1749056392 learned_addr learned my addr 172.21.15.143:0/1749056392 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:29:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.688+0000 7fd7c359e640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T19:29:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.688+0000 7fd7c359e640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:29:42.688 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:42.688+0000 7fd7c359e640 1 --1- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 0x7fd7b000cba0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:29:45.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 10 monclient(hunting): tick 2024-03-25T19:29:45.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:29:45.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:29:45.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:29:45.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6790/0 conn(0x7fd7c4146450 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.203:6789/0 conn(0x7fd7b0003680 legacy=0x7fd7b0010a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6789/0 conn(0x7fd7c40a9920 legacy=0x7fd7b000cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:45.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:45.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b00093f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:45.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b00125b0 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:45.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b00160c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:45.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40b2940 con 0x7fd7b00160c0 2024-03-25T19:29:45.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0010af0 con 0x7fd7b00093f0 2024-03-25T19:29:45.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0010d20 con 0x7fd7b00125b0 2024-03-25T19:29:45.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:45.467+0000 7fd7c0d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:48.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 10 monclient(hunting): tick 2024-03-25T19:29:48.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:29:48.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:29:48.467 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6790/0 conn(0x7fd7b00125b0 legacy=0x7fd7b00149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.203:6789/0 conn(0x7fd7b00093f0 legacy=0x7fd7b00102f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6789/0 conn(0x7fd7b00160c0 legacy=0x7fd7b0018550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b0006280 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b00066c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b00125b0 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0010f50 con 0x7fd7b0006280 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0008400 con 0x7fd7b00066c0 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0008630 con 0x7fd7b00125b0 2024-03-25T19:29:48.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:48.467+0000 7fd7c0d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 10 monclient(hunting): tick 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6790/0 conn(0x7fd7b00125b0 legacy=0x7fd7b0018550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.203:6789/0 conn(0x7fd7b00066c0 legacy=0x7fd7b0006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6789/0 conn(0x7fd7b0006280 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:51.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b00089f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:51.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b0008f20 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:51.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b0010540 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:51.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0008860 con 0x7fd7b00089f0 2024-03-25T19:29:51.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b00175f0 con 0x7fd7b0010540 2024-03-25T19:29:51.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0017820 con 0x7fd7b0008f20 2024-03-25T19:29:51.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:51.468+0000 7fd7c0d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:54.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 10 monclient(hunting): tick 2024-03-25T19:29:54.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:29:54.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 1 monclient(hunting): continuing hunt 2024-03-25T19:29:54.468 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6790/0 conn(0x7fd7b0008f20 legacy=0x7fd7b00149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.203:6789/0 conn(0x7fd7b0010540 legacy=0x7fd7b0006aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6789/0 conn(0x7fd7b00089f0 legacy=0x7fd7b000fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b00115a0 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b0011a20 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b0010540 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b0017a50 con 0x7fd7b0011a20 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b000fc60 con 0x7fd7b0010540 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b000fdf0 con 0x7fd7b00115a0 2024-03-25T19:29:54.469 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:54.468+0000 7fd7c0d99640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:56.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.634+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093975508 0 0) 0x7fd7b0017a50 con 0x7fd7b0011a20 2024-03-25T19:29:56.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.634+0000 7fd7c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.634+0000 7fd7c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.634+0000 7fd7c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.634+0000 7fd7c1d9b640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:29:56.634 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.634+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7ac003520 con 0x7fd7b0011a20 2024-03-25T19:29:56.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 957734190 0 0) 0x7fd7ac003520 con 0x7fd7b0011a20 2024-03-25T19:29:56.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6790/0 conn(0x7fd7b00115a0 legacy=0x7fd7b0005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.203:6789/0 conn(0x7fd7b0010540 legacy=0x7fd7b0006aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:29:56.689 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 1 monclient: found mon.noname-a 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b0010020 con 0x7fd7b0011a20 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd7b8004070 con 0x7fd7b0011a20 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c913d640 5 monclient: authenticate success, global_id 4101 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c913d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c1d9b640 10 monclient: dump: 2024-03-25T19:29:56.690 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:29:56.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.689+0000 7fd7c913d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.744+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7b80041d0 con 0x7fd7b0011a20 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.744+0000 7fd7c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.744+0000 7fd7c1d9b640 1 -- 172.21.15.143:0/1749056392 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd7b8004620 con 0x7fd7b0011a20 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.744+0000 7fd7c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.744+0000 7fd7c1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.744+0000 7fd7c1d9b640 10 monclient: dump: 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:56.744 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.744+0000 7fd7c913d640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:56.745 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.744+0000 7fd7c913d640 4 set_mon_vals no callback set 2024-03-25T19:29:56.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.745+0000 7fd7c913d640 10 monclient: shutdown 2024-03-25T19:29:56.747 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.745+0000 7fd7c913d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.745+0000 7fd7c913d640 1 -- 172.21.15.143:0/1749056392 >> v1:172.21.15.143:6789/0 conn(0x7fd7b0011a20 legacy=0x7fd7b0011e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.745+0000 7fd7c913d640 1 -- 172.21.15.143:0/1749056392 shutdown_connections 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.745+0000 7fd7c913d640 1 -- 172.21.15.143:0/1749056392 >> 172.21.15.143:0/1749056392 conn(0x7fd7c40b7f40 msgr2=0x7fd7c40b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.745+0000 7fd7c913d640 1 -- 172.21.15.143:0/1749056392 shutdown_connections 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.745+0000 7fd7c913d640 1 -- 172.21.15.143:0/1749056392 wait complete. 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 10 monclient: build_initial_monmap 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 10 monclient: monmap: 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:56.748 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 1 librados: starting msgr at 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 1 librados: starting objecter 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 1 Processor -- start 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 1 -- start start 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 1 librados: setting wanted keys 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 1 librados: calling monclient init 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.746+0000 7fd7c913d640 10 monclient: init 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b0010540 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b00115a0 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b0011a20 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:56.749 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40acea0 con 0x7fd7b0010540 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40ad110 con 0x7fd7b0011a20 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40ad3f0 con 0x7fd7b00115a0 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.747+0000 7fd7c913d640 10 monclient(hunting): authenticate will time out at 6013.091212s 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.748+0000 7fd7c359e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd7b0011a20 0x7fd7c41e2c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:36822/0 (socket says 172.21.15.143:36822) 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.748+0000 7fd7c359e640 1 -- 172.21.15.143:0/394055744 learned_addr learned my addr 172.21.15.143:0/394055744 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.748+0000 7fd7ab7fe640 10 client.?.objecter ms_handle_connect 0x7fd7b0011a20 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.748+0000 7fd7ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.749+0000 7fd7ab7fe640 10 client.?.objecter ms_handle_connect 0x7fd7b00115a0 2024-03-25T19:29:56.750 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.750+0000 7fd7ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.758+0000 7fd7ab7fe640 10 client.?.objecter ms_handle_connect 0x7fd7b0010540 2024-03-25T19:29:56.757 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.758+0000 7fd7ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:56.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.761+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258793387 0 0) 0x7fd7c40acea0 con 0x7fd7b0010540 2024-03-25T19:29:56.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.761+0000 7fd7ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:56.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.761+0000 7fd7ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:56.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.761+0000 7fd7ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:56.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.761+0000 7fd7ab7fe640 10 monclient(hunting): my global_id is 4110 2024-03-25T19:29:56.761 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.761+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b00101b0 con 0x7fd7b0010540 2024-03-25T19:29:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.780+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3771520905 0 0) 0x7fd7b00101b0 con 0x7fd7b0010540 2024-03-25T19:29:56.779 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.780+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7c40acea0 con 0x7fd7b0010540 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.780+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd7b40041c0 con 0x7fd7b0010540 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.780+0000 7fd7ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.780+0000 7fd7ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.780+0000 7fd7ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:29:56.780 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.780+0000 7fd7ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3905890414 0 0) 0x7fd7c40acea0 con 0x7fd7b0010540 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 >> v1:172.21.15.143:6790/0 conn(0x7fd7b00115a0 legacy=0x7fd7c41d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 >> v1:172.21.15.203:6789/0 conn(0x7fd7b0011a20 legacy=0x7fd7c41e2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 1 monclient: found mon.a 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4141930 con 0x7fd7b0010540 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:56.843 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 5 monclient: authenticate success, global_id 4110 2024-03-25T19:29:56.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 10 monclient: _renew_subs 2024-03-25T19:29:56.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:56.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c4138800 con 0x7fd7b0010540 2024-03-25T19:29:56.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:56.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 10 monclient: _renew_subs 2024-03-25T19:29:56.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:56.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c4144600 con 0x7fd7b0010540 2024-03-25T19:29:56.844 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.842+0000 7fd7c913d640 1 librados: init done 2024-03-25T19:29:56.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.843+0000 7fd7c913d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.844+0000 7fd7c913d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:56.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.844+0000 7fd7c913d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:56.845 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:56.844+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c4144600 con 0x7fd7b0010540 2024-03-25T19:29:57.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7b4004390 con 0x7fd7b0010540 2024-03-25T19:29:57.059 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd7b40047e0 con 0x7fd7b0010540 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 10 monclient: dump: 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:29:57.060 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd7b4004d90 con 0x7fd7b0010540 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7c159a640 4 set_mon_vals no callback set 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1499430107 0 0) 0x7fd7b4006520 con 0x7fd7b0010540 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 10 client.4110.objecter ms_dispatch 0x7fd7c4000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd7b4006960 con 0x7fd7b0010540 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:57.061 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.059+0000 7fd7ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:57.147 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.148+0000 7fd7c913d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:29:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.148+0000 7fd7c913d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:29:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.148+0000 7fd7c913d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:57.148 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.148+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd7c4144600 con 0x7fd7b0010540 2024-03-25T19:29:57.339 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:57.340+0000 7fc0f5871040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:29:57.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.393+0000 7fd7ab7fe640 1 -- 172.21.15.143:0/394055744 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fd7b4005120 con 0x7fd7b0010540 2024-03-25T19:29:57.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.393+0000 7fd7ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:29:57.393 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.393+0000 7fd7ab7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:29:57.393 INFO:teuthology.orchestra.run.smithi143.stderr:adjusted tunables profile to default 2024-03-25T19:29:57.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 10 librados: watch_flush enter 2024-03-25T19:29:57.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 10 librados: watch_flush exit 2024-03-25T19:29:57.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T19:29:57.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 10 client.4110.objecter successfully canceled tick 2024-03-25T19:29:57.394 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 10 monclient: shutdown 2024-03-25T19:29:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 >> v1:172.21.15.143:6789/0 conn(0x7fd7b0010540 legacy=0x7fd7c41380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 shutdown_connections 2024-03-25T19:29:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 >> 172.21.15.143:0/394055744 conn(0x7fd7c40b7f40 msgr2=0x7fd7c41395a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 shutdown_connections 2024-03-25T19:29:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 1 -- 172.21.15.143:0/394055744 wait complete. 2024-03-25T19:29:57.395 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:29:57.394+0000 7fd7c913d640 1 librados: shutdown 2024-03-25T19:29:57.399 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:57.399+0000 7fc0f5871040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:29:57.405 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:29:57.405 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:29:57.405 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:57.405 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:29:57.432 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:57.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:29:57.499 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:57.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:29:57.568 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-25T19:29:57.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:29:57.618 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:57.618+0000 7f577c0c5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:29:57.636 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:57.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:29:57.664 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:57.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:29:57.681 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:57.681+0000 7f577c0c5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:29:57.707 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:57.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:29:57.774 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-25T19:29:57.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:29:57.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 822e7ad9-f685-4d65-981e-8366577fa717 0 2024-03-25T19:29:57.972 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:57.972+0000 7fc0f5871040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:29:57.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.982+0000 7faf7e152640 10 monclient: get_monmap_and_config 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.982+0000 7faf7e152640 10 monclient: build_initial_monmap 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.982+0000 7faf7e152640 10 monclient: monmap: 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:57.983278+0000 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:57.983278+0000 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:29:57.982 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 1 Processor -- start 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 1 -- start start 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 10 monclient: init 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 10 monclient(hunting): picked mon.noname-b con 0x7faf780b4700 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 10 monclient(hunting): picked mon.noname-c con 0x7faf78142a20 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 10 monclient(hunting): picked mon.noname-a con 0x7faf780a96b0 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf780a9d60 con 0x7faf780a96b0 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7813ff10 con 0x7faf78142a20 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf780aa1f0 con 0x7faf780b4700 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:57.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.983+0000 7faf7e152640 10 monclient(hunting): authenticate will time out at 6029.233388s 2024-03-25T19:29:57.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.984+0000 7faf77fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7faf780a96b0 0x7faf7813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55478/0 (socket says 172.21.15.203:55478) 2024-03-25T19:29:57.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.984+0000 7faf77fff640 1 -- 172.21.15.203:0/2254680490 learned_addr learned my addr 172.21.15.203:0/2254680490 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:57.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.986+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829560453 0 0) 0x7faf7813ff10 con 0x7faf78142a20 2024-03-25T19:29:57.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.986+0000 7faf767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:57.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.986+0000 7faf767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:57.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.986+0000 7faf767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:57.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.986+0000 7faf767fc640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:29:57.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.986+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf60003520 con 0x7faf78142a20 2024-03-25T19:29:57.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622854189 0 0) 0x7faf780aa1f0 con 0x7faf780b4700 2024-03-25T19:29:57.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:57.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:57.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:57.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:29:57.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf7813ff10 con 0x7faf780b4700 2024-03-25T19:29:57.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510810310 0 0) 0x7faf780a9d60 con 0x7faf780a96b0 2024-03-25T19:29:57.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:57.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:57.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:57.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:29:57.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.987+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf780aa1f0 con 0x7faf780a96b0 2024-03-25T19:29:57.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2102406722 0 0) 0x7faf780aa1f0 con 0x7faf780a96b0 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 >> v1:172.21.15.143:6790/0 conn(0x7faf780b4700 legacy=0x7faf78142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 >> v1:172.21.15.203:6789/0 conn(0x7faf78142a20 legacy=0x7faf7813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 1 monclient: found mon.noname-a 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf780b4ae0 con 0x7faf780a96b0 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf7e152640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:29:57.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.990+0000 7faf7e152640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.991+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf70001870 con 0x7faf780a96b0 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.991+0000 7faf767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.991+0000 7faf767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.991+0000 7faf767fc640 10 monclient: dump: 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:57.992 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.991+0000 7faf7e152640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf70001b40 con 0x7faf780a96b0 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf767fc640 1 -- 172.21.15.203:0/2254680490 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf70001ed0 con 0x7faf780a96b0 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 4 set_mon_vals no callback set 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf767fc640 10 monclient: dump: 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:57.993 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 10 monclient: shutdown 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 1 -- 172.21.15.203:0/2254680490 >> v1:172.21.15.143:6789/0 conn(0x7faf780a96b0 legacy=0x7faf7813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 1 -- 172.21.15.203:0/2254680490 shutdown_connections 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 1 -- 172.21.15.203:0/2254680490 >> 172.21.15.203:0/2254680490 conn(0x7faf780668e0 msgr2=0x7faf78066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 1 -- 172.21.15.203:0/2254680490 shutdown_connections 2024-03-25T19:29:57.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.992+0000 7faf7e152640 1 -- 172.21.15.203:0/2254680490 wait complete. 2024-03-25T19:29:57.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 10 monclient: build_initial_monmap 2024-03-25T19:29:57.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 10 monclient: monmap: 2024-03-25T19:29:57.998 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:57.998 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:57.998 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T19:29:57.998 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T19:29:57.998 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:57.999 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T19:29:57.999 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:57.999 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:29:57.999 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:29:57.999 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:57.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 1 librados: starting msgr at 2024-03-25T19:29:58.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 1 librados: starting objecter 2024-03-25T19:29:58.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 1 Processor -- start 2024-03-25T19:29:58.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 1 -- start start 2024-03-25T19:29:58.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 1 librados: setting wanted keys 2024-03-25T19:29:58.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 1 librados: calling monclient init 2024-03-25T19:29:58.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.993+0000 7faf7e152640 10 monclient: init 2024-03-25T19:29:58.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 10 monclient(hunting): picked mon.noname-c con 0x7faf780a96b0 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 10 monclient(hunting): picked mon.noname-a con 0x7faf780b4700 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 10 monclient(hunting): picked mon.noname-b con 0x7faf78142a20 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf781d7600 con 0x7faf780b4700 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7813ccf0 con 0x7faf78142a20 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf781d4fe0 con 0x7faf780a96b0 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf7e152640 10 monclient(hunting): authenticate will time out at 6029.244031s 2024-03-25T19:29:58.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf76ffd640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7faf780b4700 0x7faf7813bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55486/0 (socket says 172.21.15.203:55486) 2024-03-25T19:29:58.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.994+0000 7faf76ffd640 1 -- 172.21.15.203:0/1097813359 learned_addr learned my addr 172.21.15.203:0/1097813359 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:58.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.995+0000 7faf5ffff640 10 client.?.objecter ms_handle_connect 0x7faf780b4700 2024-03-25T19:29:58.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.995+0000 7faf5ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.997+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315192558 0 0) 0x7faf781d7600 con 0x7faf780b4700 2024-03-25T19:29:58.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.997+0000 7faf5ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.997+0000 7faf5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.997+0000 7faf5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.997+0000 7faf5ffff640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:29:58.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:57.997+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf64004eb0 con 0x7faf780b4700 2024-03-25T19:29:58.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.001+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1935579316 0 0) 0x7faf64004eb0 con 0x7faf780b4700 2024-03-25T19:29:58.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.001+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf781d7600 con 0x7faf780b4700 2024-03-25T19:29:58.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.001+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf6c001910 con 0x7faf780b4700 2024-03-25T19:29:58.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.001+0000 7faf5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.001+0000 7faf5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:58.005 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.001+0000 7faf5ffff640 10 monclient(hunting): dump: 2024-03-25T19:29:58.005 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.005 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.005 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.006 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.006 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.006 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.006 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.006 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.006 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.006 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.001+0000 7faf5ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2384594821 0 0) 0x7faf781d7600 con 0x7faf780b4700 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 >> v1:172.21.15.143:6790/0 conn(0x7faf780a96b0 legacy=0x7faf781d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 >> v1:172.21.15.203:6789/0 conn(0x7faf78142a20 legacy=0x7faf781e1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 1 monclient: found mon.a 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf781d8230 con 0x7faf780b4700 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 10 monclient: _finish_auth 0 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:58.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:29:58.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 10 monclient: _renew_subs 2024-03-25T19:29:58.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7813c3e0 con 0x7faf780b4700 2024-03-25T19:29:58.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:58.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 10 monclient: _renew_subs 2024-03-25T19:29:58.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7813b6f0 con 0x7faf780b4700 2024-03-25T19:29:58.008 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.004+0000 7faf7e152640 1 librados: init done 2024-03-25T19:29:58.009 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.006+0000 7faf7e152640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.009 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.006+0000 7faf7e152640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.009 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.006+0000 7faf7e152640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.009 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.006+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7804f730 con 0x7faf780b4700 2024-03-25T19:29:58.009 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf6c001a70 con 0x7faf780b4700 2024-03-25T19:29:58.009 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf6c0038f0 con 0x7faf780b4700 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 10 monclient: dump: 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.010 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faf6c003ea0 con 0x7faf780b4700 2024-03-25T19:29:58.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf75ffb640 4 set_mon_vals no callback set 2024-03-25T19:29:58.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 955988813 0 0) 0x7faf6c006630 con 0x7faf780b4700 2024-03-25T19:29:58.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 10 client.4119.objecter ms_dispatch 0x7faf78000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:29:58.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:29:58.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:29:58.011 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.008+0000 7faf5ffff640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:29:58.028 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.028+0000 7fc0f5871040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:29:58.075 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.076+0000 7fc0f5871040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:29:58.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.104+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faf6c004150 con 0x7faf780b4700 2024-03-25T19:29:58.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.104+0000 7faf5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.104 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.104+0000 7faf5ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:58.130 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.131+0000 7fc0f5871040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:29:58.172 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.173+0000 7fc0f5871040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:29:58.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.191+0000 7faf7e152640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "822e7ad9-f685-4d65-981e-8366577fa717", "id": 0}] 2024-03-25T19:29:58.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.191+0000 7faf7e152640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "822e7ad9-f685-4d65-981e-8366577fa717", "id": 0}] 2024-03-25T19:29:58.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.191+0000 7faf7e152640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.191+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd new", "uuid": "822e7ad9-f685-4d65-981e-8366577fa717", "id": 0} v 0) v1 -- 0x7faf7813d600 con 0x7faf780b4700 2024-03-25T19:29:58.244 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.245+0000 7f577c0c5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:29:58.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.294+0000 7faf5ffff640 1 -- 172.21.15.203:0/1097813359 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "822e7ad9-f685-4d65-981e-8366577fa717", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1998205607 0 732707515) 0x7faf6c0043c0 con 0x7faf780b4700 2024-03-25T19:29:58.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.294+0000 7faf5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "822e7ad9-f685-4d65-981e-8366577fa717", "id": 0}] 2024-03-25T19:29:58.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.294+0000 7faf5ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:58.294 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-03-25T19:29:58.297 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.295+0000 7fc0f5871040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:29:58.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 10 librados: watch_flush enter 2024-03-25T19:29:58.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 10 librados: watch_flush exit 2024-03-25T19:29:58.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:29:58.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:29:58.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 10 monclient: shutdown 2024-03-25T19:29:58.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:58.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 >> v1:172.21.15.143:6789/0 conn(0x7faf780b4700 legacy=0x7faf7813bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 shutdown_connections 2024-03-25T19:29:58.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 >> 172.21.15.203:0/1097813359 conn(0x7faf780668e0 msgr2=0x7faf7813f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:58.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 shutdown_connections 2024-03-25T19:29:58.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 1 -- 172.21.15.203:0/1097813359 wait complete. 2024-03-25T19:29:58.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.295+0000 7faf7e152640 1 librados: shutdown 2024-03-25T19:29:58.301 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.302+0000 7f577c0c5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:29:58.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 388a4708-38b7-4278-a660-a79b90807cbe 1 2024-03-25T19:29:58.350 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.351+0000 7f577c0c5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:29:58.366 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.367+0000 7fc0f5871040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:29:58.406 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.406+0000 7f577c0c5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:29:58.408 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.408+0000 7fed328d9640 10 monclient: get_monmap_and_config 2024-03-25T19:29:58.408 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.408+0000 7fed328d9640 10 monclient: build_initial_monmap 2024-03-25T19:29:58.408 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.408+0000 7fed328d9640 10 monclient: monmap: 2024-03-25T19:29:58.408 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:58.408 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.409 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:58.409506+0000 2024-03-25T19:29:58.409 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:58.409506+0000 2024-03-25T19:29:58.409 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:58.409 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.409 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:58.409 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:29:58.409 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:29:58.409 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 1 Processor -- start 2024-03-25T19:29:58.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 1 -- start start 2024-03-25T19:29:58.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 10 monclient: init 2024-03-25T19:29:58.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:58.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 10 monclient(hunting): picked mon.noname-c con 0x7fed2c0b4700 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 10 monclient(hunting): picked mon.noname-b con 0x7fed2c142a20 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 10 monclient(hunting): picked mon.noname-a con 0x7fed2c0a96b0 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c0a9d60 con 0x7fed2c0a96b0 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c13ff10 con 0x7fed2c0b4700 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c0aa1f0 con 0x7fed2c142a20 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.409+0000 7fed328d9640 10 monclient(hunting): authenticate will time out at 6029.659535s 2024-03-25T19:29:58.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.410+0000 7fed30e76640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fed2c0a96b0 0x7fed2c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55494/0 (socket says 172.21.15.203:55494) 2024-03-25T19:29:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.410+0000 7fed30e76640 1 -- 172.21.15.203:0/3454095418 learned_addr learned my addr 172.21.15.203:0/3454095418 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942140377 0 0) 0x7fed2c13ff10 con 0x7fed2c0b4700 2024-03-25T19:29:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:29:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed14003520 con 0x7fed2c0b4700 2024-03-25T19:29:58.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248563583 0 0) 0x7fed2c0a9d60 con 0x7fed2c0a96b0 2024-03-25T19:29:58.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:29:58.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.412+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2c13ff10 con 0x7fed2c0a96b0 2024-03-25T19:29:58.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.413+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061070348 0 0) 0x7fed2c0aa1f0 con 0x7fed2c142a20 2024-03-25T19:29:58.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.413+0000 7fed2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.413+0000 7fed2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.413+0000 7fed2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.413+0000 7fed2affd640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:29:58.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.413+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2c0a9d60 con 0x7fed2c142a20 2024-03-25T19:29:58.418 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.412+0000 7fc0f5871040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531122795 0 0) 0x7fed2c13ff10 con 0x7fed2c0a96b0 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 >> v1:172.21.15.143:6790/0 conn(0x7fed2c142a20 legacy=0x7fed2c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 >> v1:172.21.15.203:6789/0 conn(0x7fed2c0b4700 legacy=0x7fed2c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 1 monclient: found mon.noname-a 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c0b4ae0 con 0x7fed2c0a96b0 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fed240019f0 con 0x7fed2c0a96b0 2024-03-25T19:29:58.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed2affd640 10 monclient: dump: 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed328d9640 5 monclient: authenticate success, global_id 4122 2024-03-25T19:29:58.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.416+0000 7fed328d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.418+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed24001b50 con 0x7fed2c0a96b0 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.418+0000 7fed2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.418+0000 7fed2affd640 1 -- 172.21.15.203:0/3454095418 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fed24001fa0 con 0x7fed2c0a96b0 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.418+0000 7fed2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.418+0000 7fed2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.418+0000 7fed2affd640 10 monclient: dump: 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.427 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.418+0000 7fed328d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.418+0000 7fed328d9640 4 set_mon_vals no callback set 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 10 monclient: shutdown 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 1 -- 172.21.15.203:0/3454095418 >> v1:172.21.15.143:6789/0 conn(0x7fed2c0a96b0 legacy=0x7fed2c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 1 -- 172.21.15.203:0/3454095418 shutdown_connections 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 1 -- 172.21.15.203:0/3454095418 >> 172.21.15.203:0/3454095418 conn(0x7fed2c0668e0 msgr2=0x7fed2c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 1 -- 172.21.15.203:0/3454095418 shutdown_connections 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 1 -- 172.21.15.203:0/3454095418 wait complete. 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 10 monclient: build_initial_monmap 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.419+0000 7fed328d9640 10 monclient: monmap: 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:58.428 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 librados: starting msgr at 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 librados: starting objecter 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 Processor -- start 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 -- start start 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 librados: setting wanted keys 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 librados: calling monclient init 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 10 monclient: init 2024-03-25T19:29:58.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 10 monclient(hunting): picked mon.noname-b con 0x7fed2c0a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 10 monclient(hunting): picked mon.noname-c con 0x7fed2c0b4700 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 10 monclient(hunting): picked mon.noname-a con 0x7fed2c142a20 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c0ade20 con 0x7fed2c142a20 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c1d7490 con 0x7fed2c0a96b0 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2c1d3fd0 con 0x7fed2c0b4700 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.420+0000 7fed328d9640 10 monclient(hunting): authenticate will time out at 6029.670519s 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.421+0000 7fed30e76640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fed2c142a20 0x7fed2c0ac380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55506/0 (socket says 172.21.15.203:55506) 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.421+0000 7fed30e76640 1 -- 172.21.15.203:0/2406454753 learned_addr learned my addr 172.21.15.203:0/2406454753 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.421+0000 7fed28ff9640 10 client.?.objecter ms_handle_connect 0x7fed2c142a20 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.421+0000 7fed28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.421+0000 7fed28ff9640 10 client.?.objecter ms_handle_connect 0x7fed2c0a96b0 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.421+0000 7fed28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.421+0000 7fed28ff9640 10 client.?.objecter ms_handle_connect 0x7fed2c0b4700 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.421+0000 7fed28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135477409 0 0) 0x7fed2c0ade20 con 0x7fed2c142a20 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed18004eb0 con 0x7fed2c142a20 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410027072 0 0) 0x7fed2c1d7490 con 0x7fed2c0a96b0 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:29:58.431 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2c0ade20 con 0x7fed2c0a96b0 2024-03-25T19:29:58.432 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909031991 0 0) 0x7fed2c1d3fd0 con 0x7fed2c0b4700 2024-03-25T19:29:58.432 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.432 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.432 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.423+0000 7fed28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.432 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.424+0000 7fed28ff9640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:29:58.432 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.424+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2c1d7490 con 0x7fed2c0b4700 2024-03-25T19:29:58.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.429+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 295139817 0 0) 0x7fed18004eb0 con 0x7fed2c142a20 2024-03-25T19:29:58.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.429+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed2c1d3fd0 con 0x7fed2c142a20 2024-03-25T19:29:58.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.429+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fed24001f60 con 0x7fed2c142a20 2024-03-25T19:29:58.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.429+0000 7fed28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.429+0000 7fed28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.429+0000 7fed28ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.435 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.436 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.436 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.436 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.436 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.429+0000 7fed28ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:58.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 901585937 0 0) 0x7fed2c1d3fd0 con 0x7fed2c142a20 2024-03-25T19:29:58.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 >> v1:172.21.15.143:6790/0 conn(0x7fed2c0b4700 legacy=0x7fed2c1d4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 >> v1:172.21.15.203:6789/0 conn(0x7fed2c0a96b0 legacy=0x7fed2c1d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:58.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 1 monclient: found mon.a 2024-03-25T19:29:58.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:58.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c1d4160 con 0x7fed2c142a20 2024-03-25T19:29:58.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:58.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:58.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:58.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.432+0000 7fed328d9640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:29:58.438 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 10 monclient: _renew_subs 2024-03-25T19:29:58.439 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.439 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2c1d80c0 con 0x7fed2c142a20 2024-03-25T19:29:58.439 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:58.439 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 10 monclient: _renew_subs 2024-03-25T19:29:58.439 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.439 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2c1d4e70 con 0x7fed2c142a20 2024-03-25T19:29:58.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 1 librados: init done 2024-03-25T19:29:58.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.433+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2c13ae30 con 0x7fed2c142a20 2024-03-25T19:29:58.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed24004e10 con 0x7fed2c142a20 2024-03-25T19:29:58.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:58.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fed24004f80 con 0x7fed2c142a20 2024-03-25T19:29:58.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:58.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 10 monclient: dump: 2024-03-25T19:29:58.443 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.443 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.443 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.443 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.443 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.443 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.444 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.444 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.444 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.444 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fed24005470 con 0x7fed2c142a20 2024-03-25T19:29:58.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed2a7fc640 4 set_mon_vals no callback set 2024-03-25T19:29:58.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 770748082 0 0) 0x7fed24006880 con 0x7fed2c142a20 2024-03-25T19:29:58.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 10 client.4125.objecter ms_dispatch 0x7fed2c000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:29:58.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:29:58.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:29:58.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.437+0000 7fed28ff9640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:29:58.452 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.453+0000 7f577c0c5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:29:58.468 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.469+0000 7fc0f5871040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:29:58.516 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.516+0000 7fc0f5871040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:29:58.532 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.532+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fed24006b40 con 0x7fed2c142a20 2024-03-25T19:29:58.532 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.532+0000 7fed28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.532 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.532+0000 7fed28ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:58.570 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.570+0000 7f577c0c5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:29:58.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.604+0000 7fed328d9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "388a4708-38b7-4278-a660-a79b90807cbe", "id": 1}] 2024-03-25T19:29:58.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.604+0000 7fed328d9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "388a4708-38b7-4278-a660-a79b90807cbe", "id": 1}] 2024-03-25T19:29:58.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.604+0000 7fed328d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.604+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd new", "uuid": "388a4708-38b7-4278-a660-a79b90807cbe", "id": 1} v 0) v1 -- 0x7fed2c04f730 con 0x7fed2c142a20 2024-03-25T19:29:58.642 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.642+0000 7f577c0c5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:29:58.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.663+0000 7fed28ff9640 1 -- 172.21.15.203:0/2406454753 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "388a4708-38b7-4278-a660-a79b90807cbe", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3945159414 0 940483276) 0x7fed24005780 con 0x7fed2c142a20 2024-03-25T19:29:58.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.663+0000 7fed28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "388a4708-38b7-4278-a660-a79b90807cbe", "id": 1}] 2024-03-25T19:29:58.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.663+0000 7fed28ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:58.663 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-03-25T19:29:58.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 10 librados: watch_flush enter 2024-03-25T19:29:58.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 10 librados: watch_flush exit 2024-03-25T19:29:58.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:29:58.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:29:58.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 10 monclient: shutdown 2024-03-25T19:29:58.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:58.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 >> v1:172.21.15.143:6789/0 conn(0x7fed2c142a20 legacy=0x7fed2c0ac380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 shutdown_connections 2024-03-25T19:29:58.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 >> 172.21.15.203:0/2406454753 conn(0x7fed2c0668e0 msgr2=0x7fed2c0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:58.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 shutdown_connections 2024-03-25T19:29:58.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 1 -- 172.21.15.203:0/2406454753 wait complete. 2024-03-25T19:29:58.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.664+0000 7fed328d9640 1 librados: shutdown 2024-03-25T19:29:58.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 5b4e989e-4066-411c-8d76-23d9e8f27d83 2 2024-03-25T19:29:58.692 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.693+0000 7f577c0c5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:29:58.750 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.751+0000 7f577c0c5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:29:58.792 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.793+0000 7fb32f0f1640 10 monclient: get_monmap_and_config 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.793+0000 7fb32f0f1640 10 monclient: build_initial_monmap 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.793+0000 7fb32f0f1640 10 monclient: monmap: 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:58.794092+0000 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:58.794092+0000 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:29:58.793 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.793+0000 7fb32f0f1640 1 Processor -- start 2024-03-25T19:29:58.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 1 -- start start 2024-03-25T19:29:58.795 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 10 monclient: init 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 10 monclient(hunting): picked mon.noname-c con 0x7fb3280b4700 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 10 monclient(hunting): picked mon.noname-b con 0x7fb328142a20 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 10 monclient(hunting): picked mon.noname-a con 0x7fb3280a96b0 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280a9d60 con 0x7fb3280a96b0 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32813ff10 con 0x7fb3280b4700 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280aa1f0 con 0x7fb328142a20 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.794+0000 7fb32f0f1640 10 monclient(hunting): authenticate will time out at 6030.044208s 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.795+0000 7fb32ce8d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb3280b4700 0x7fb328142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59350/0 (socket says 172.21.15.203:59350) 2024-03-25T19:29:58.796 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.795+0000 7fb32ce8d640 1 -- 172.21.15.203:0/739237828 learned_addr learned my addr 172.21.15.203:0/739237828 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:58.797 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579559122 0 0) 0x7fb3280a9d60 con 0x7fb3280a96b0 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb318003520 con 0x7fb3280a96b0 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404189748 0 0) 0x7fb32813ff10 con 0x7fb3280b4700 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.797+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3280a9d60 con 0x7fb3280b4700 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.798+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746418940 0 0) 0x7fb3280aa1f0 con 0x7fb328142a20 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.798+0000 7fb3277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.798 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.798+0000 7fb3277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.798+0000 7fb3277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.798+0000 7fb3277fe640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:29:58.799 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.798+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb32813ff10 con 0x7fb328142a20 2024-03-25T19:29:58.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1430261605 0 0) 0x7fb3280a9d60 con 0x7fb3280b4700 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 >> v1:172.21.15.143:6790/0 conn(0x7fb328142a20 legacy=0x7fb32813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 >> v1:172.21.15.143:6789/0 conn(0x7fb3280a96b0 legacy=0x7fb32813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 1 monclient: found mon.noname-c 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3280b4ae0 con 0x7fb3280b4700 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:58.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:58.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb32f0f1640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:29:58.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb31c001670 con 0x7fb3280b4700 2024-03-25T19:29:58.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb32f0f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:58.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:58.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb3277fe640 10 monclient: dump: 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.803 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.804 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.800+0000 7fb32f0f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:58.804 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:58.801+0000 7f577c0c5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:29:58.826 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb31c0017d0 con 0x7fb3280b4700 2024-03-25T19:29:58.826 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb3277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb3277fe640 1 -- 172.21.15.203:0/739237828 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb31c001be0 con 0x7fb3280b4700 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb32f0f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb32f0f1640 4 set_mon_vals no callback set 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb3277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb3277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb3277fe640 10 monclient: dump: 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.827 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.828 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.828 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.828 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.828 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb32f0f1640 10 monclient: shutdown 2024-03-25T19:29:58.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb32f0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:58.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb32f0f1640 1 -- 172.21.15.203:0/739237828 >> v1:172.21.15.203:6789/0 conn(0x7fb3280b4700 legacy=0x7fb328142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb32f0f1640 1 -- 172.21.15.203:0/739237828 shutdown_connections 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb32f0f1640 1 -- 172.21.15.203:0/739237828 >> 172.21.15.203:0/739237828 conn(0x7fb3280668e0 msgr2=0x7fb328066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.827+0000 7fb32f0f1640 1 -- 172.21.15.203:0/739237828 shutdown_connections 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.828+0000 7fb32f0f1640 1 -- 172.21.15.203:0/739237828 wait complete. 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.828+0000 7fb32f0f1640 10 monclient: build_initial_monmap 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.828+0000 7fb32f0f1640 10 monclient: monmap: 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:29:58.832 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 librados: starting msgr at 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 librados: starting objecter 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 Processor -- start 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 -- start start 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 librados: setting wanted keys 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 librados: calling monclient init 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 10 monclient: init 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 10 monclient(hunting): picked mon.noname-c con 0x7fb3280a96b0 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 10 monclient(hunting): picked mon.noname-b con 0x7fb3280b4700 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 10 monclient(hunting): picked mon.noname-a con 0x7fb328142a20 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280acb90 con 0x7fb328142a20 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280add20 con 0x7fb3280b4700 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280ae000 con 0x7fb3280a96b0 2024-03-25T19:29:58.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.829+0000 7fb32f0f1640 10 monclient(hunting): authenticate will time out at 6030.079522s 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.830+0000 7fb327fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb3280b4700 0x7fb3280ae420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59356/0 (socket says 172.21.15.203:59356) 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.830+0000 7fb327fff640 1 -- 172.21.15.203:0/3588583951 learned_addr learned my addr 172.21.15.203:0/3588583951 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.830+0000 7fb3257fa640 10 client.?.objecter ms_handle_connect 0x7fb3280b4700 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.830+0000 7fb3257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.830+0000 7fb3257fa640 10 client.?.objecter ms_handle_connect 0x7fb328142a20 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.830+0000 7fb3257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.831+0000 7fb3257fa640 10 client.?.objecter ms_handle_connect 0x7fb3280a96b0 2024-03-25T19:29:58.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.831+0000 7fb3257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010999402 0 0) 0x7fb3280add20 con 0x7fb3280b4700 2024-03-25T19:29:58.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:29:58.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb30c004eb0 con 0x7fb3280b4700 2024-03-25T19:29:58.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183774729 0 0) 0x7fb3280acb90 con 0x7fb328142a20 2024-03-25T19:29:58.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:29:58.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.832+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3280add20 con 0x7fb328142a20 2024-03-25T19:29:58.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.833+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560456101 0 0) 0x7fb3280ae000 con 0x7fb3280a96b0 2024-03-25T19:29:58.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.833+0000 7fb3257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.838 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.833+0000 7fb3257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.838 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.833+0000 7fb3257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.838 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.833+0000 7fb3257fa640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:29:58.838 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.833+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3280acb90 con 0x7fb3280a96b0 2024-03-25T19:29:58.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 831052756 0 0) 0x7fb30c004eb0 con 0x7fb3280b4700 2024-03-25T19:29:58.839 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3280ae000 con 0x7fb3280b4700 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb314001a10 con 0x7fb3280b4700 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 10 monclient(hunting): dump: 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.840 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 704797757 0 0) 0x7fb3280add20 con 0x7fb328142a20 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb30c004eb0 con 0x7fb328142a20 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb320001a70 con 0x7fb328142a20 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.837+0000 7fb3257fa640 10 monclient(hunting): dump: 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.841 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.838+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2473058486 0 0) 0x7fb3280acb90 con 0x7fb3280a96b0 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.838+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3280add20 con 0x7fb3280a96b0 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.838+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb31c001ae0 con 0x7fb3280a96b0 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.838+0000 7fb3257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.838+0000 7fb3257fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.838+0000 7fb3257fa640 10 monclient(hunting): dump: 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.842 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.843 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.843 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.843 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.843 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1868799437 0 0) 0x7fb30c004eb0 con 0x7fb328142a20 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 >> v1:172.21.15.143:6790/0 conn(0x7fb3280a96b0 legacy=0x7fb3280ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 >> v1:172.21.15.203:6789/0 conn(0x7fb3280b4700 legacy=0x7fb3280ae420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 10 monclient: _finish_hunting 0 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 1 monclient: found mon.a 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3280aeb30 con 0x7fb328142a20 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 10 monclient: _finish_auth 0 2024-03-25T19:29:58.846 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 10 monclient: _check_auth_tickets 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb3257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb32f0f1640 5 monclient: authenticate success, global_id 4131 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb32f0f1640 10 monclient: _renew_subs 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb32f0f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.840+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3280ab9a0 con 0x7fb328142a20 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 10 monclient: _renew_subs 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3281356c0 con 0x7fb328142a20 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 1 librados: init done 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.841+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3281359c0 con 0x7fb328142a20 2024-03-25T19:29:58.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb320001f60 con 0x7fb328142a20 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb320003990 con 0x7fb328142a20 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 10 monclient: dump: 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:58.848 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:58.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb320003e80 con 0x7fb328142a20 2024-03-25T19:29:58.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb326ffd640 4 set_mon_vals no callback set 2024-03-25T19:29:58.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2236431495 0 0) 0x7fb320005320 con 0x7fb328142a20 2024-03-25T19:29:58.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 10 client.4131.objecter ms_dispatch 0x7fb328000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:29:58.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:29:58.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:29:58.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.845+0000 7fb3257fa640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T19:29:58.914 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:58.914+0000 7fc0f5871040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:29:58.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.938+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb320005010 con 0x7fb328142a20 2024-03-25T19:29:58.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.938+0000 7fb3257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:58.938+0000 7fb3257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:59.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.010+0000 7fb32f0f1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5b4e989e-4066-411c-8d76-23d9e8f27d83", "id": 2}] 2024-03-25T19:29:59.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.010+0000 7fb32f0f1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5b4e989e-4066-411c-8d76-23d9e8f27d83", "id": 2}] 2024-03-25T19:29:59.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.010+0000 7fb32f0f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:59.010 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.010+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5b4e989e-4066-411c-8d76-23d9e8f27d83", "id": 2} v 0) v1 -- 0x7fb3280acec0 con 0x7fb328142a20 2024-03-25T19:29:59.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.040+0000 7fb3257fa640 1 -- 172.21.15.203:0/3588583951 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5b4e989e-4066-411c-8d76-23d9e8f27d83", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2410453282 0 216599125) 0x7fb320004200 con 0x7fb328142a20 2024-03-25T19:29:59.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.040+0000 7fb3257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5b4e989e-4066-411c-8d76-23d9e8f27d83", "id": 2}] 2024-03-25T19:29:59.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.040+0000 7fb3257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:59.040 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 10 librados: watch_flush enter 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 10 librados: watch_flush exit 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 10 client.4131.objecter successfully canceled tick 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 10 monclient: shutdown 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 >> v1:172.21.15.143:6789/0 conn(0x7fb328142a20 legacy=0x7fb3281e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 shutdown_connections 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 >> 172.21.15.203:0/3588583951 conn(0x7fb3280668e0 msgr2=0x7fb3280b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 shutdown_connections 2024-03-25T19:29:59.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.041+0000 7fb32f0f1640 1 -- 172.21.15.203:0/3588583951 wait complete. 2024-03-25T19:29:59.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.042+0000 7fb32f0f1640 1 librados: shutdown 2024-03-25T19:29:59.052 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 0490eb42-6979-4362-9b8c-efaeb3e7bb54 3 2024-03-25T19:29:59.063 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:59.064+0000 7fc0f5871040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:29:59.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.153+0000 7faf91f23640 10 monclient: get_monmap_and_config 2024-03-25T19:29:59.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.153+0000 7faf91f23640 10 monclient: build_initial_monmap 2024-03-25T19:29:59.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.153+0000 7faf91f23640 10 monclient: monmap: 2024-03-25T19:29:59.153 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:59.154765+0000 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:59.154765+0000 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:29:59.154 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.154+0000 7faf91f23640 1 Processor -- start 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.154+0000 7faf91f23640 1 -- start start 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.154+0000 7faf91f23640 10 monclient: init 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 10 monclient(hunting): picked mon.noname-a con 0x7faf8c0b4700 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 10 monclient(hunting): picked mon.noname-c con 0x7faf8c142a20 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 10 monclient(hunting): picked mon.noname-b con 0x7faf8c0a96b0 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c0a9d60 con 0x7faf8c0b4700 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c13ff10 con 0x7faf8c142a20 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c0aa1f0 con 0x7faf8c0a96b0 2024-03-25T19:29:59.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:59.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.155+0000 7faf91f23640 10 monclient(hunting): authenticate will time out at 6030.405087s 2024-03-25T19:29:59.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.156+0000 7faf83fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7faf8c142a20 0x7faf8c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59366/0 (socket says 172.21.15.203:59366) 2024-03-25T19:29:59.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.156+0000 7faf8b7fe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7faf8c0b4700 0x7faf8c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55526/0 (socket says 172.21.15.203:55526) 2024-03-25T19:29:59.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.156+0000 7faf83fff640 1 -- 172.21.15.203:0/3035609685 learned_addr learned my addr 172.21.15.203:0/3035609685 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:59.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210725830 0 0) 0x7faf8c13ff10 con 0x7faf8c142a20 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf78003520 con 0x7faf8c142a20 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860013029 0 0) 0x7faf8c0a9d60 con 0x7faf8c0b4700 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:29:59.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.158+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c13ff10 con 0x7faf8c0b4700 2024-03-25T19:29:59.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.159+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193753294 0 0) 0x7faf8c0aa1f0 con 0x7faf8c0a96b0 2024-03-25T19:29:59.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.159+0000 7faf8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.159+0000 7faf8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.159+0000 7faf8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.159+0000 7faf8affd640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:29:59.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.159+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c0a9d60 con 0x7faf8c0a96b0 2024-03-25T19:29:59.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 315263521 0 0) 0x7faf78003520 con 0x7faf8c142a20 2024-03-25T19:29:59.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 >> v1:172.21.15.143:6790/0 conn(0x7faf8c0a96b0 legacy=0x7faf8c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 >> v1:172.21.15.143:6789/0 conn(0x7faf8c0b4700 legacy=0x7faf8c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:59.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 1 monclient: found mon.noname-c 2024-03-25T19:29:59.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0b4ae0 con 0x7faf8c142a20 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf74001690 con 0x7faf8c142a20 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf91f23640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf91f23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf8affd640 10 monclient: dump: 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.162 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.163 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.163 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.163 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.163 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.163 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.163 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.163 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.161+0000 7faf91f23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:59.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf740017f0 con 0x7faf8c142a20 2024-03-25T19:29:59.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:59.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf8affd640 1 -- 172.21.15.203:0/3035609685 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf74001c00 con 0x7faf8c142a20 2024-03-25T19:29:59.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:59.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 4 set_mon_vals no callback set 2024-03-25T19:29:59.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf8affd640 10 monclient: dump: 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 10 monclient: shutdown 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:59.165 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 1 -- 172.21.15.203:0/3035609685 >> v1:172.21.15.203:6789/0 conn(0x7faf8c142a20 legacy=0x7faf8c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 1 -- 172.21.15.203:0/3035609685 shutdown_connections 2024-03-25T19:29:59.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 1 -- 172.21.15.203:0/3035609685 >> 172.21.15.203:0/3035609685 conn(0x7faf8c0668e0 msgr2=0x7faf8c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:59.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 1 -- 172.21.15.203:0/3035609685 shutdown_connections 2024-03-25T19:29:59.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.164+0000 7faf91f23640 1 -- 172.21.15.203:0/3035609685 wait complete. 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 10 monclient: build_initial_monmap 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 10 monclient: monmap: 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:29:59.168 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 1 librados: starting msgr at 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 1 librados: starting objecter 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 1 Processor -- start 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 1 -- start start 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 1 librados: setting wanted keys 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 1 librados: calling monclient init 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.165+0000 7faf91f23640 10 monclient: init 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 10 monclient(hunting): picked mon.noname-b con 0x7faf8c0a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 10 monclient(hunting): picked mon.noname-c con 0x7faf8c0b4700 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 10 monclient(hunting): picked mon.noname-a con 0x7faf8c142a20 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c0ae240 con 0x7faf8c142a20 2024-03-25T19:29:59.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c1d3fd0 con 0x7faf8c0a96b0 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8c1d4160 con 0x7faf8c0b4700 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.166+0000 7faf91f23640 10 monclient(hunting): authenticate will time out at 6030.416089s 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.167+0000 7faf8bfff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7faf8c142a20 0x7faf8c1daaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55530/0 (socket says 172.21.15.203:55530) 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.167+0000 7faf8bfff640 1 -- 172.21.15.203:0/3752870840 learned_addr learned my addr 172.21.15.203:0/3752870840 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.167+0000 7faf88ff9640 10 client.?.objecter ms_handle_connect 0x7faf8c142a20 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.167+0000 7faf88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.167+0000 7faf88ff9640 10 client.?.objecter ms_handle_connect 0x7faf8c0a96b0 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.167+0000 7faf88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.167+0000 7faf88ff9640 10 client.?.objecter ms_handle_connect 0x7faf8c0b4700 2024-03-25T19:29:59.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.167+0000 7faf88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:59.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40951007 0 0) 0x7faf8c0ae240 con 0x7faf8c142a20 2024-03-25T19:29:59.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c004eb0 con 0x7faf8c142a20 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190786596 0 0) 0x7faf8c1d3fd0 con 0x7faf8c0a96b0 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c0ae240 con 0x7faf8c0a96b0 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186647929 0 0) 0x7faf8c1d4160 con 0x7faf8c0b4700 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:29:59.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.169+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c1d3fd0 con 0x7faf8c0b4700 2024-03-25T19:29:59.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.173+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3806838518 0 0) 0x7faf6c004eb0 con 0x7faf8c142a20 2024-03-25T19:29:59.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.173+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf8c1d4160 con 0x7faf8c142a20 2024-03-25T19:29:59.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.173+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf84001d10 con 0x7faf8c142a20 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.173+0000 7faf88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.173+0000 7faf88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.173+0000 7faf88ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.173+0000 7faf88ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2157366282 0 0) 0x7faf8c0ae240 con 0x7faf8c0a96b0 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf6c004eb0 con 0x7faf8c0a96b0 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf7c001970 con 0x7faf8c0a96b0 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.175 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.176 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3926845258 0 0) 0x7faf8c1d3fd0 con 0x7faf8c0b4700 2024-03-25T19:29:59.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf8c0ae240 con 0x7faf8c0b4700 2024-03-25T19:29:59.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf74001b00 con 0x7faf8c0b4700 2024-03-25T19:29:59.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.174+0000 7faf88ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1389863751 0 0) 0x7faf8c1d4160 con 0x7faf8c142a20 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 >> v1:172.21.15.143:6790/0 conn(0x7faf8c0b4700 legacy=0x7faf8c1d4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 >> v1:172.21.15.203:6789/0 conn(0x7faf8c0a96b0 legacy=0x7faf8c0ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 1 monclient: found mon.a 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0acee0 con 0x7faf8c142a20 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 5 monclient: authenticate success, global_id 4137 2024-03-25T19:29:59.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 10 monclient: _renew_subs 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8c1d4d40 con 0x7faf8c142a20 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 10 monclient: _renew_subs 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8c1db3e0 con 0x7faf8c142a20 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.177+0000 7faf91f23640 1 librados: init done 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.178+0000 7faf91f23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.178+0000 7faf91f23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.178+0000 7faf91f23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:59.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.178+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8c0ad130 con 0x7faf8c142a20 2024-03-25T19:29:59.185 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:59.186+0000 7fc0f5871040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:29:59.204 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:59.205+0000 7f577c0c5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:29:59.232 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:59.233+0000 7fc0f5871040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.274+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf84003990 con 0x7faf8c142a20 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.274+0000 7faf88ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.274+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf84003d60 con 0x7faf8c142a20 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.274+0000 7faf88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.274+0000 7faf88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.274+0000 7faf88ff9640 10 monclient: dump: 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.274 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.274+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faf84004310 con 0x7faf8c142a20 2024-03-25T19:29:59.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.274+0000 7faf8a7fc640 4 set_mon_vals no callback set 2024-03-25T19:29:59.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.275+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1711309496 0 0) 0x7faf840069d0 con 0x7faf8c142a20 2024-03-25T19:29:59.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.275+0000 7faf88ff9640 10 client.4137.objecter ms_dispatch 0x7faf8c000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:29:59.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.275+0000 7faf88ff9640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:29:59.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.275+0000 7faf88ff9640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:29:59.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.275+0000 7faf88ff9640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T19:29:59.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.275+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faf84006540 con 0x7faf8c142a20 2024-03-25T19:29:59.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.275+0000 7faf88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:59.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.275+0000 7faf88ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:59.341 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:59.342+0000 7f577c0c5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:29:59.346 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:59.347+0000 7fc0f5871040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:29:59.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.348+0000 7faf91f23640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0490eb42-6979-4362-9b8c-efaeb3e7bb54", "id": 3}] 2024-03-25T19:29:59.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.348+0000 7faf91f23640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0490eb42-6979-4362-9b8c-efaeb3e7bb54", "id": 3}] 2024-03-25T19:29:59.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.348+0000 7faf91f23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:29:59.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.348+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0490eb42-6979-4362-9b8c-efaeb3e7bb54", "id": 3} v 0) v1 -- 0x7faf8c04f730 con 0x7faf8c142a20 2024-03-25T19:29:59.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.383+0000 7faf88ff9640 1 -- 172.21.15.203:0/3752870840 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0490eb42-6979-4362-9b8c-efaeb3e7bb54", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3198023176 0 525046306) 0x7faf840047a0 con 0x7faf8c142a20 2024-03-25T19:29:59.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.383+0000 7faf88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0490eb42-6979-4362-9b8c-efaeb3e7bb54", "id": 3}] 2024-03-25T19:29:59.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.383+0000 7faf88ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:59.384 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-03-25T19:29:59.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 10 librados: watch_flush enter 2024-03-25T19:29:59.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 10 librados: watch_flush exit 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 10 client.4137.objecter successfully canceled tick 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 10 monclient: shutdown 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 >> v1:172.21.15.143:6789/0 conn(0x7faf8c142a20 legacy=0x7faf8c1daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 shutdown_connections 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.384+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 >> 172.21.15.203:0/3752870840 conn(0x7faf8c0668e0 msgr2=0x7faf8c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.385+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 shutdown_connections 2024-03-25T19:29:59.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.385+0000 7faf91f23640 1 -- 172.21.15.203:0/3752870840 wait complete. 2024-03-25T19:29:59.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.385+0000 7faf91f23640 1 librados: shutdown 2024-03-25T19:29:59.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new bf6a03da-5706-4ec1-aca2-b21e11e3c8ac 4 2024-03-25T19:29:59.452 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:59.453+0000 7fc0f5871040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:29:59.483 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:59.484+0000 7f577c0c5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.496+0000 7f8719c8d640 10 monclient: get_monmap_and_config 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.496+0000 7f8719c8d640 10 monclient: build_initial_monmap 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.496+0000 7f8719c8d640 10 monclient: monmap: 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:59.497480+0000 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:59.497480+0000 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:29:59.496 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:29:59.497 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.497+0000 7f8719c8d640 1 Processor -- start 2024-03-25T19:29:59.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.497+0000 7f8719c8d640 1 -- start start 2024-03-25T19:29:59.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.497+0000 7f8719c8d640 10 monclient: init 2024-03-25T19:29:59.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:59.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:59.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f87140b4700 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:59.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f8714142a20 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:59.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f87140a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:59.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87140a9d60 con 0x7f8714142a20 2024-03-25T19:29:59.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871413ff10 con 0x7f87140a96b0 2024-03-25T19:29:59.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87140aa1f0 con 0x7f87140b4700 2024-03-25T19:29:59.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:59.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8719c8d640 10 monclient(hunting): authenticate will time out at 6030.747742s 2024-03-25T19:29:59.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8713fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f87140a96b0 0x7f871413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59376/0 (socket says 172.21.15.203:59376) 2024-03-25T19:29:59.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.498+0000 7f8713fff640 1 -- 172.21.15.203:0/1443883240 learned_addr learned my addr 172.21.15.203:0/1443883240 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885025717 0 0) 0x7f87140a9d60 con 0x7f8714142a20 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8700003520 con 0x7f8714142a20 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470658032 0 0) 0x7f87140aa1f0 con 0x7f87140b4700 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87140a9d60 con 0x7f87140b4700 2024-03-25T19:29:59.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40831642 0 0) 0x7f871413ff10 con 0x7f87140a96b0 2024-03-25T19:29:59.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:29:59.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.501+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87140aa1f0 con 0x7f87140a96b0 2024-03-25T19:29:59.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157026381 0 0) 0x7f8700003520 con 0x7f8714142a20 2024-03-25T19:29:59.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 >> v1:172.21.15.143:6790/0 conn(0x7f87140b4700 legacy=0x7f8714142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 >> v1:172.21.15.203:6789/0 conn(0x7f87140a96b0 legacy=0x7f871413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:59.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 1 monclient: found mon.noname-a 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87140b4ae0 con 0x7f8714142a20 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f86fc001690 con 0x7f8714142a20 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8712ffd640 10 monclient: dump: 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.505 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.506 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.506 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.506 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.506 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.506 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8719c8d640 5 monclient: authenticate success, global_id 4140 2024-03-25T19:29:59.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.504+0000 7f8719c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.506+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86fc0017f0 con 0x7f8714142a20 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.506+0000 7f8712ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.506+0000 7f8712ffd640 1 -- 172.21.15.203:0/1443883240 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f86fc001c00 con 0x7f8714142a20 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.506+0000 7f8719c8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.506+0000 7f8719c8d640 4 set_mon_vals no callback set 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.506+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.506+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.506+0000 7f8712ffd640 10 monclient: dump: 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.509 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 10 monclient: shutdown 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 1 -- 172.21.15.203:0/1443883240 >> v1:172.21.15.143:6789/0 conn(0x7f8714142a20 legacy=0x7f871413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 1 -- 172.21.15.203:0/1443883240 shutdown_connections 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 1 -- 172.21.15.203:0/1443883240 >> 172.21.15.203:0/1443883240 conn(0x7f87140668e0 msgr2=0x7f8714066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 1 -- 172.21.15.203:0/1443883240 shutdown_connections 2024-03-25T19:29:59.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 1 -- 172.21.15.203:0/1443883240 wait complete. 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 10 monclient: build_initial_monmap 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.507+0000 7f8719c8d640 10 monclient: monmap: 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 librados: starting msgr at 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 librados: starting objecter 2024-03-25T19:29:59.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 Processor -- start 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 -- start start 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 librados: setting wanted keys 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 librados: calling monclient init 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 10 monclient: init 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f87140a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f87140b4700 addr v1:172.21.15.143:6789/0 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f8714142a20 addr v1:172.21.15.143:6790/0 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871413b5d0 con 0x7f87140b4700 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871413c480 con 0x7f87140a96b0 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871413b850 con 0x7f8714142a20 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.508+0000 7f8719c8d640 10 monclient(hunting): authenticate will time out at 6030.758408s 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.509+0000 7f870bfff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f87140b4700 0x7f87141dabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55544/0 (socket says 172.21.15.203:55544) 2024-03-25T19:29:59.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.509+0000 7f870bfff640 1 -- 172.21.15.203:0/2086839975 learned_addr learned my addr 172.21.15.203:0/2086839975 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:29:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.509+0000 7f8710ff9640 10 client.?.objecter ms_handle_connect 0x7f87140b4700 2024-03-25T19:29:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.509+0000 7f8710ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.509+0000 7f8710ff9640 10 client.?.objecter ms_handle_connect 0x7f87140a96b0 2024-03-25T19:29:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.509+0000 7f8710ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.509+0000 7f8710ff9640 10 client.?.objecter ms_handle_connect 0x7f8714142a20 2024-03-25T19:29:59.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.510+0000 7f8710ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474773815 0 0) 0x7f871413b5d0 con 0x7f87140b4700 2024-03-25T19:29:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:29:59.514 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f4004eb0 con 0x7f87140b4700 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973993032 0 0) 0x7f871413c480 con 0x7f87140a96b0 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.511+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871413b5d0 con 0x7f87140a96b0 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.513+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145109630 0 0) 0x7f871413b850 con 0x7f8714142a20 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.513+0000 7f8710ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.513+0000 7f8710ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.513+0000 7f8710ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.513+0000 7f8710ff9640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:29:59.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.513+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871413c480 con 0x7f8714142a20 2024-03-25T19:29:59.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3822107835 0 0) 0x7f86f4004eb0 con 0x7f87140b4700 2024-03-25T19:29:59.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f871413b850 con 0x7f87140b4700 2024-03-25T19:29:59.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f86fc001bc0 con 0x7f87140b4700 2024-03-25T19:29:59.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:59.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:59.518 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1099549794 0 0) 0x7f871413c480 con 0x7f8714142a20 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86f4004eb0 con 0x7f8714142a20 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f870c001bd0 con 0x7f8714142a20 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1399206981 0 0) 0x7f871413b5d0 con 0x7f87140a96b0 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f871413c480 con 0x7f87140a96b0 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8704001990 con 0x7f87140a96b0 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.518+0000 7f8710ff9640 10 monclient(hunting): dump: 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.521 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2708999682 0 0) 0x7f86f4004eb0 con 0x7f8714142a20 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 >> v1:172.21.15.203:6789/0 conn(0x7f87140a96b0 legacy=0x7f871413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 >> v1:172.21.15.143:6789/0 conn(0x7f87140b4700 legacy=0x7f87141dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 1 monclient: found mon.c 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87141d4fe0 con 0x7f8714142a20 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8710ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 5 monclient: authenticate success, global_id 4133 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 10 monclient: _renew_subs 2024-03-25T19:29:59.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87141e2ca0 con 0x7f8714142a20 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 10 monclient: _renew_subs 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87141e3010 con 0x7f8714142a20 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.522+0000 7f8719c8d640 1 librados: init done 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.523+0000 7f8719c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.523+0000 7f8719c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.523+0000 7f8719c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:29:59.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.523+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871404f730 con 0x7f8714142a20 2024-03-25T19:29:59.539 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:59.540+0000 7f577c0c5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:29:59.615 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:59.616+0000 7fc0f5871040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:29:59.651 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:59.651+0000 7f577c0c5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:29:59.659 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:59.659+0000 7fc0f5871040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:29:59.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.738+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f870c001fe0 con 0x7f8714142a20 2024-03-25T19:29:59.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.738+0000 7f8710ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:59.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.738+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f870c004d00 con 0x7f8714142a20 2024-03-25T19:29:59.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.738+0000 7f8710ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.738+0000 7f8710ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.738+0000 7f8710ff9640 10 monclient: dump: 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.738+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f870c0051f0 con 0x7f8714142a20 2024-03-25T19:29:59.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.738+0000 7f87127fc640 4 set_mon_vals no callback set 2024-03-25T19:29:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.741+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2742214174 0 0) 0x7f870c006830 con 0x7f8714142a20 2024-03-25T19:29:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.741+0000 7f8710ff9640 10 client.4133.objecter ms_dispatch 0x7f8714000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:29:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.741+0000 7f8710ff9640 3 client.4133.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:29:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.741+0000 7f8710ff9640 3 client.4133.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:29:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.741+0000 7f8710ff9640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T19:29:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.741+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f870c006b80 con 0x7f8714142a20 2024-03-25T19:29:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.741+0000 7f8710ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:59.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.741+0000 7f8710ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:59.747 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:59.748+0000 7f577c0c5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:29:59.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.814+0000 7f8719c8d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bf6a03da-5706-4ec1-aca2-b21e11e3c8ac", "id": 4}] 2024-03-25T19:29:59.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.814+0000 7f8719c8d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bf6a03da-5706-4ec1-aca2-b21e11e3c8ac", "id": 4}] 2024-03-25T19:29:59.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.814+0000 7f8719c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:29:59.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.814+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd new", "uuid": "bf6a03da-5706-4ec1-aca2-b21e11e3c8ac", "id": 4} v 0) v1 -- 0x7f8714069f30 con 0x7f8714142a20 2024-03-25T19:29:59.823 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:29:59.824+0000 7fc0f5871040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:29:59.908 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:59.909+0000 7f577c0c5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:29:59.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.937+0000 7f8710ff9640 1 -- 172.21.15.203:0/2086839975 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf6a03da-5706-4ec1-aca2-b21e11e3c8ac", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 134458931 0 1697012583) 0x7f870c0063f0 con 0x7f8714142a20 2024-03-25T19:29:59.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.937+0000 7f8710ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bf6a03da-5706-4ec1-aca2-b21e11e3c8ac", "id": 4}] 2024-03-25T19:29:59.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.937+0000 7f8710ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:59.937 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-03-25T19:29:59.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 10 librados: watch_flush enter 2024-03-25T19:29:59.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 10 librados: watch_flush exit 2024-03-25T19:29:59.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 10 client.4133.objecter successfully canceled tick 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 10 monclient: shutdown 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 >> v1:172.21.15.143:6790/0 conn(0x7f8714142a20 legacy=0x7f87141db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 shutdown_connections 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 >> 172.21.15.203:0/2086839975 conn(0x7f87140668e0 msgr2=0x7f87140b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 shutdown_connections 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 1 -- 172.21.15.203:0/2086839975 wait complete. 2024-03-25T19:29:59.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:29:59.938+0000 7f8719c8d640 1 librados: shutdown 2024-03-25T19:29:59.948 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 44b74cce-f2bf-4da3-bf62-8768784cad61 5 2024-03-25T19:29:59.953 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:29:59.954+0000 7f577c0c5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.048+0000 7fe17359d640 10 monclient: get_monmap_and_config 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.048+0000 7fe17359d640 10 monclient: build_initial_monmap 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.048+0000 7fe17359d640 10 monclient: monmap: 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:30:00.049534+0000 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:30:00.049534+0000 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:00.048 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.049 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:00.049 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:00.049 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:00.049 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.049+0000 7fe17359d640 1 Processor -- start 2024-03-25T19:30:00.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.049+0000 7fe17359d640 1 -- start start 2024-03-25T19:30:00.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.049+0000 7fe17359d640 10 monclient: init 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.049+0000 7fe17359d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 10 monclient(hunting): picked mon.noname-a con 0x7fe16c0a9920 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 10 monclient(hunting): picked mon.noname-c con 0x7fe16c0b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 10 monclient(hunting): picked mon.noname-b con 0x7fe16c146450 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c06b100 con 0x7fe16c0a9920 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c0b2940 con 0x7fe16c0b45a0 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c13fec0 con 0x7fe16c146450 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe17359d640 10 monclient(hunting): authenticate will time out at 6031.299660s 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe171339640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fe16c0a9920 0x7fe16c13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55546/0 (socket says 172.21.15.203:55546) 2024-03-25T19:30:00.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.050+0000 7fe171339640 1 -- 172.21.15.203:0/1560764940 learned_addr learned my addr 172.21.15.203:0/1560764940 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:30:00.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183342606 0 0) 0x7fe16c13fec0 con 0x7fe16c146450 2024-03-25T19:30:00.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe158003520 con 0x7fe16c146450 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339777165 0 0) 0x7fe16c0b2940 con 0x7fe16c0b45a0 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c13fec0 con 0x7fe16c0b45a0 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561770573 0 0) 0x7fe16c06b100 con 0x7fe16c0a9920 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:30:00.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.053+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c0b2940 con 0x7fe16c0a9920 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 123134796 0 0) 0x7fe158003520 con 0x7fe16c146450 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 >> v1:172.21.15.203:6789/0 conn(0x7fe16c0b45a0 legacy=0x7fe16c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 >> v1:172.21.15.143:6789/0 conn(0x7fe16c0a9920 legacy=0x7fe16c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 1 monclient: found mon.noname-b 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c140050 con 0x7fe16c146450 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 10 monclient: _finish_auth 0 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe164001a10 con 0x7fe16c146450 2024-03-25T19:30:00.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe163fff640 10 monclient: dump: 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.057 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe17359d640 5 monclient: authenticate success, global_id 4136 2024-03-25T19:30:00.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.056+0000 7fe17359d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.058+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe164001b70 con 0x7fe16c146450 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.058+0000 7fe163fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.058+0000 7fe163fff640 1 -- 172.21.15.203:0/1560764940 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe164001de0 con 0x7fe16c146450 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.058+0000 7fe17359d640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.058+0000 7fe17359d640 4 set_mon_vals no callback set 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.058+0000 7fe163fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.058+0000 7fe163fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.058+0000 7fe163fff640 10 monclient: dump: 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.061 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 10 monclient: shutdown 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 1 -- 172.21.15.203:0/1560764940 >> v1:172.21.15.143:6790/0 conn(0x7fe16c146450 legacy=0x7fe16c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 1 -- 172.21.15.203:0/1560764940 shutdown_connections 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 1 -- 172.21.15.203:0/1560764940 >> 172.21.15.203:0/1560764940 conn(0x7fe16c0b7f40 msgr2=0x7fe16c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:00.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 1 -- 172.21.15.203:0/1560764940 shutdown_connections 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 1 -- 172.21.15.203:0/1560764940 wait complete. 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 10 monclient: build_initial_monmap 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.059+0000 7fe17359d640 10 monclient: monmap: 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 librados: starting msgr at 2024-03-25T19:30:00.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 librados: starting objecter 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 Processor -- start 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 -- start start 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 librados: setting wanted keys 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 librados: calling monclient init 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 10 monclient: init 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 10 monclient(hunting): picked mon.noname-a con 0x7fe16c0a9920 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 10 monclient(hunting): picked mon.noname-b con 0x7fe16c0b45a0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 10 monclient(hunting): picked mon.noname-c con 0x7fe16c146450 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c1d7370 con 0x7fe16c0a9920 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c137e30 con 0x7fe16c0b45a0 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c145630 con 0x7fe16c146450 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:00.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.060+0000 7fe17359d640 10 monclient(hunting): authenticate will time out at 6031.310354s 2024-03-25T19:30:00.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.061+0000 7fe170b38640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe16c0b45a0 0x7fe16c0b12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59398/0 (socket says 172.21.15.203:59398) 2024-03-25T19:30:00.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.061+0000 7fe170b38640 1 -- 172.21.15.203:0/689466313 learned_addr learned my addr 172.21.15.203:0/689466313 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:30:00.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.061+0000 7fe161ffb640 10 client.?.objecter ms_handle_connect 0x7fe16c0b45a0 2024-03-25T19:30:00.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.061+0000 7fe161ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.061+0000 7fe161ffb640 10 client.?.objecter ms_handle_connect 0x7fe16c146450 2024-03-25T19:30:00.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.061+0000 7fe161ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.061+0000 7fe161ffb640 10 client.?.objecter ms_handle_connect 0x7fe16c0a9920 2024-03-25T19:30:00.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.061+0000 7fe161ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.063+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629405489 0 0) 0x7fe16c145630 con 0x7fe16c146450 2024-03-25T19:30:00.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.063+0000 7fe161ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.063+0000 7fe161ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.063+0000 7fe161ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.063+0000 7fe161ffb640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.063+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe150004eb0 con 0x7fe16c146450 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585045236 0 0) 0x7fe16c1d7370 con 0x7fe16c0a9920 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c145630 con 0x7fe16c0a9920 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307791056 0 0) 0x7fe16c137e30 con 0x7fe16c0b45a0 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:30:00.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.064+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c1d7370 con 0x7fe16c0b45a0 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.068+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2473575165 0 0) 0x7fe150004eb0 con 0x7fe16c146450 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.068+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe16c137e30 con 0x7fe16c146450 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.068+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe164001e80 con 0x7fe16c146450 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.068+0000 7fe161ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.068+0000 7fe161ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.068+0000 7fe161ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.068 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.068+0000 7fe161ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:00.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.069+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2867291806 0 0) 0x7fe16c145630 con 0x7fe16c0a9920 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.069+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe150004eb0 con 0x7fe16c0a9920 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.069+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe168001870 con 0x7fe16c0a9920 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.069+0000 7fe161ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.069+0000 7fe161ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.069+0000 7fe161ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.070 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.071 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.071 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.071 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.070+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3610507061 0 0) 0x7fe16c1d7370 con 0x7fe16c0b45a0 2024-03-25T19:30:00.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.070+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe16c145630 con 0x7fe16c0b45a0 2024-03-25T19:30:00.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.070+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe15c001930 con 0x7fe16c0b45a0 2024-03-25T19:30:00.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1284892699 0 0) 0x7fe16c137e30 con 0x7fe16c146450 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 >> v1:172.21.15.203:6789/0 conn(0x7fe16c0b45a0 legacy=0x7fe16c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 >> v1:172.21.15.143:6789/0 conn(0x7fe16c0a9920 legacy=0x7fe16c144f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:00.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 1 monclient: found mon.c 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c0ac900 con 0x7fe16c146450 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe161ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe17359d640 5 monclient: authenticate success, global_id 4139 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe17359d640 10 monclient: _renew_subs 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe17359d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.071+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c1417d0 con 0x7fe16c146450 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.072+0000 7fe17359d640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.072+0000 7fe17359d640 10 monclient: _renew_subs 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.072+0000 7fe17359d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:00.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.072+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c0b1c00 con 0x7fe16c146450 2024-03-25T19:30:00.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.072+0000 7fe17359d640 1 librados: init done 2024-03-25T19:30:00.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.073+0000 7fe17359d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.073+0000 7fe17359d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.073+0000 7fe17359d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:00.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.073+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c04f730 con 0x7fe16c146450 2024-03-25T19:30:00.121 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:30:00.122+0000 7f577c0c5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:30:00.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.171+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe164002020 con 0x7fe16c146450 2024-03-25T19:30:00.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.171+0000 7fe161ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:00.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.171+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe164003dc0 con 0x7fe16c146450 2024-03-25T19:30:00.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.171+0000 7fe161ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.171+0000 7fe161ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.171+0000 7fe161ffb640 10 monclient: dump: 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.171+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe1640042b0 con 0x7fe16c146450 2024-03-25T19:30:00.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.171+0000 7fe1637fe640 4 set_mon_vals no callback set 2024-03-25T19:30:00.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.172+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 895888207 0 0) 0x7fe1640058a0 con 0x7fe16c146450 2024-03-25T19:30:00.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.172+0000 7fe161ffb640 10 client.4139.objecter ms_dispatch 0x7fe16c000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:30:00.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.172+0000 7fe161ffb640 3 client.4139.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:30:00.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.172+0000 7fe161ffb640 3 client.4139.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:30:00.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.172+0000 7fe161ffb640 20 client.4139.objecter dump_active .. 0 homeless 2024-03-25T19:30:00.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.172+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe164005b90 con 0x7fe16c146450 2024-03-25T19:30:00.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.172+0000 7fe161ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.172+0000 7fe161ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:00.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.243+0000 7fe17359d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "44b74cce-f2bf-4da3-bf62-8768784cad61", "id": 5}] 2024-03-25T19:30:00.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.243+0000 7fe17359d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "44b74cce-f2bf-4da3-bf62-8768784cad61", "id": 5}] 2024-03-25T19:30:00.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.243+0000 7fe17359d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:00.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.243+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd new", "uuid": "44b74cce-f2bf-4da3-bf62-8768784cad61", "id": 5} v 0) v1 -- 0x7fe16c069f30 con 0x7fe16c146450 2024-03-25T19:30:00.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.301+0000 7fe161ffb640 1 -- 172.21.15.203:0/689466313 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "44b74cce-f2bf-4da3-bf62-8768784cad61", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 880346459 0 1988412176) 0x7fe1640053f0 con 0x7fe16c146450 2024-03-25T19:30:00.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.301+0000 7fe161ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "44b74cce-f2bf-4da3-bf62-8768784cad61", "id": 5}] 2024-03-25T19:30:00.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.301+0000 7fe161ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:00.302 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-03-25T19:30:00.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 10 librados: watch_flush enter 2024-03-25T19:30:00.304 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 10 librados: watch_flush exit 2024-03-25T19:30:00.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 20 client.4139.objecter shutdown clearing up homeless session... 2024-03-25T19:30:00.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 10 client.4139.objecter successfully canceled tick 2024-03-25T19:30:00.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 10 monclient: shutdown 2024-03-25T19:30:00.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:00.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 >> v1:172.21.15.143:6790/0 conn(0x7fe16c146450 legacy=0x7fe16c1d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 shutdown_connections 2024-03-25T19:30:00.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 >> 172.21.15.203:0/689466313 conn(0x7fe16c0b7f40 msgr2=0x7fe16c13f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:00.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 shutdown_connections 2024-03-25T19:30:00.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 1 -- 172.21.15.203:0/689466313 wait complete. 2024-03-25T19:30:00.306 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.302+0000 7fe17359d640 1 librados: shutdown 2024-03-25T19:30:00.313 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 7a5ddf86-73e1-4c36-96da-8b4707ebb1c1 6 2024-03-25T19:30:00.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.455+0000 7f09d0d39640 10 monclient: get_monmap_and_config 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.455+0000 7f09d0d39640 10 monclient: build_initial_monmap 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.455+0000 7f09d0d39640 10 monclient: monmap: 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:30:00.456531+0000 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:30:00.456531+0000 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.456+0000 7f09d0d39640 1 Processor -- start 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.456+0000 7f09d0d39640 1 -- start start 2024-03-25T19:30:00.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.456+0000 7f09d0d39640 10 monclient: init 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.456+0000 7f09d0d39640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc0b4700 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc142a20 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc0a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0a9d60 con 0x7f09cc142a20 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc13ff10 con 0x7f09cc0a96b0 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0aa1f0 con 0x7f09cc0b4700 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09d0d39640 10 monclient(hunting): authenticate will time out at 6031.706646s 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09c3fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f09cc142a20 0x7f09cc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55558/0 (socket says 172.21.15.203:55558) 2024-03-25T19:30:00.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.457+0000 7f09c3fff640 1 -- 172.21.15.203:0/151833394 learned_addr learned my addr 172.21.15.203:0/151833394 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:30:00.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053678246 0 0) 0x7f09cc0aa1f0 con 0x7f09cc0b4700 2024-03-25T19:30:00.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:30:00.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b8003520 con 0x7f09cc0b4700 2024-03-25T19:30:00.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667709201 0 0) 0x7f09cc0a9d60 con 0x7f09cc142a20 2024-03-25T19:30:00.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:30:00.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.460+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc0aa1f0 con 0x7f09cc142a20 2024-03-25T19:30:00.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.461+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959169027 0 0) 0x7f09cc13ff10 con 0x7f09cc0a96b0 2024-03-25T19:30:00.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.461+0000 7f09c9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.461+0000 7f09c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.461+0000 7f09c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.461+0000 7f09c9d9b640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:30:00.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.461+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc0a9d60 con 0x7f09cc0a96b0 2024-03-25T19:30:00.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1419440693 0 0) 0x7f09cc0aa1f0 con 0x7f09cc142a20 2024-03-25T19:30:00.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 >> v1:172.21.15.143:6790/0 conn(0x7f09cc0b4700 legacy=0x7f09cc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 >> v1:172.21.15.203:6789/0 conn(0x7f09cc0a96b0 legacy=0x7f09cc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 1 monclient: found mon.noname-a 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc0b4ae0 con 0x7f09cc142a20 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f09b4001670 con 0x7f09cc142a20 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09c9d9b640 10 monclient: dump: 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.464 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09d0d39640 5 monclient: authenticate success, global_id 4152 2024-03-25T19:30:00.465 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.464+0000 7f09d0d39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09b40017d0 con 0x7f09cc142a20 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09c9d9b640 1 -- 172.21.15.203:0/151833394 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f09b4001be0 con 0x7f09cc142a20 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09c9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09c9d9b640 10 monclient: dump: 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.467 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 4 set_mon_vals no callback set 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 10 monclient: shutdown 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 1 -- 172.21.15.203:0/151833394 >> v1:172.21.15.143:6789/0 conn(0x7f09cc142a20 legacy=0x7f09cc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 1 -- 172.21.15.203:0/151833394 shutdown_connections 2024-03-25T19:30:00.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 1 -- 172.21.15.203:0/151833394 >> 172.21.15.203:0/151833394 conn(0x7f09cc0668e0 msgr2=0x7f09cc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 1 -- 172.21.15.203:0/151833394 shutdown_connections 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.466+0000 7f09d0d39640 1 -- 172.21.15.203:0/151833394 wait complete. 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.467+0000 7f09d0d39640 10 monclient: build_initial_monmap 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.467+0000 7f09d0d39640 10 monclient: monmap: 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.467+0000 7f09d0d39640 1 librados: starting msgr at 2024-03-25T19:30:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.467+0000 7f09d0d39640 1 librados: starting objecter 2024-03-25T19:30:00.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.467+0000 7f09d0d39640 1 Processor -- start 2024-03-25T19:30:00.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 1 -- start start 2024-03-25T19:30:00.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 1 librados: setting wanted keys 2024-03-25T19:30:00.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 1 librados: calling monclient init 2024-03-25T19:30:00.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 10 monclient: init 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc0a96b0 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc0b4700 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc142a20 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc13c060 con 0x7f09cc0a96b0 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc13ca90 con 0x7f09cc142a20 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0aba90 con 0x7f09cc0b4700 2024-03-25T19:30:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.468+0000 7f09d0d39640 10 monclient(hunting): authenticate will time out at 6031.718078s 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.469+0000 7f09ca59c640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f09cc0a96b0 0x7f09cc13b800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55568/0 (socket says 172.21.15.203:55568) 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.469+0000 7f09ca59c640 1 -- 172.21.15.203:0/1750862359 learned_addr learned my addr 172.21.15.203:0/1750862359 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.469+0000 7f09c2ffd640 10 client.?.objecter ms_handle_connect 0x7f09cc142a20 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.469+0000 7f09c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.469+0000 7f09c2ffd640 10 client.?.objecter ms_handle_connect 0x7f09cc0a96b0 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.469+0000 7f09c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.469+0000 7f09c2ffd640 10 client.?.objecter ms_handle_connect 0x7f09cc0b4700 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.469+0000 7f09c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462446492 0 0) 0x7f09cc13ca90 con 0x7f09cc142a20 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ac004eb0 con 0x7f09cc142a20 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419466696 0 0) 0x7f09cc0aba90 con 0x7f09cc0b4700 2024-03-25T19:30:00.473 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc13ca90 con 0x7f09cc0b4700 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937007198 0 0) 0x7f09cc13c060 con 0x7f09cc0a96b0 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:30:00.474 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.471+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc0aba90 con 0x7f09cc0a96b0 2024-03-25T19:30:00.475 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.475+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3271628458 0 0) 0x7f09ac004eb0 con 0x7f09cc142a20 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.475+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09cc13c060 con 0x7f09cc142a20 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f09c4001c10 con 0x7f09cc142a20 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.476 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3778327925 0 0) 0x7f09cc13ca90 con 0x7f09cc0b4700 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09ac004eb0 con 0x7f09cc0b4700 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f09b4001ae0 con 0x7f09cc0b4700 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.476+0000 7f09c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.477 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.478 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.477+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1062142625 0 0) 0x7f09cc0aba90 con 0x7f09cc0a96b0 2024-03-25T19:30:00.478 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.477+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09cc13ca90 con 0x7f09cc0a96b0 2024-03-25T19:30:00.478 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.477+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f09bc001910 con 0x7f09cc0a96b0 2024-03-25T19:30:00.478 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.477+0000 7f09c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.478 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.477+0000 7f09c2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.477+0000 7f09c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.479 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.480 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2470292679 0 0) 0x7f09cc13c060 con 0x7f09cc142a20 2024-03-25T19:30:00.480 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 >> v1:172.21.15.143:6790/0 conn(0x7f09cc0b4700 legacy=0x7f09cc138650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 >> v1:172.21.15.143:6789/0 conn(0x7f09cc0a96b0 legacy=0x7f09cc13b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 1 monclient: found mon.b 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc13af40 con 0x7f09cc142a20 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 10 monclient: _renew_subs 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09cc13b1f0 con 0x7f09cc142a20 2024-03-25T19:30:00.481 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:00.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 10 monclient: _renew_subs 2024-03-25T19:30:00.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:00.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc1e2e40 con 0x7f09cc142a20 2024-03-25T19:30:00.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.479+0000 7f09d0d39640 1 librados: init done 2024-03-25T19:30:00.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.480+0000 7f09d0d39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.480+0000 7f09d0d39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.480+0000 7f09d0d39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:00.482 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.480+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09cc04f730 con 0x7f09cc142a20 2024-03-25T19:30:00.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.483+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09c40020e0 con 0x7f09cc142a20 2024-03-25T19:30:00.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.483+0000 7f09c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:00.483 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.483+0000 7f09c959a640 4 set_mon_vals no callback set 2024-03-25T19:30:00.648 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:30:00.648+0000 7fc0f5871040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f09c4003b30 con 0x7f09cc142a20 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 10 monclient: dump: 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:00.678 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f09c4004020 con 0x7f09cc142a20 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2964774498 0 0) 0x7f09c40058f0 con 0x7f09cc142a20 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 10 client.4150.objecter ms_dispatch 0x7f09cc000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f09c4017de0 con 0x7f09cc142a20 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:00.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.678+0000 7f09c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:00.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.752+0000 7f09d0d39640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7a5ddf86-73e1-4c36-96da-8b4707ebb1c1", "id": 6}] 2024-03-25T19:30:00.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.752+0000 7f09d0d39640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7a5ddf86-73e1-4c36-96da-8b4707ebb1c1", "id": 6}] 2024-03-25T19:30:00.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.752+0000 7f09d0d39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:00.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.752+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7a5ddf86-73e1-4c36-96da-8b4707ebb1c1", "id": 6} v 0) v1 -- 0x7f09cc069f30 con 0x7f09cc142a20 2024-03-25T19:30:00.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.933+0000 7f09c2ffd640 1 -- 172.21.15.203:0/1750862359 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7a5ddf86-73e1-4c36-96da-8b4707ebb1c1", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4136660158 0 1113811849) 0x7f09c40053d0 con 0x7f09cc142a20 2024-03-25T19:30:00.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.933+0000 7f09c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7a5ddf86-73e1-4c36-96da-8b4707ebb1c1", "id": 6}] 2024-03-25T19:30:00.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.933+0000 7f09c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:00.934 INFO:teuthology.orchestra.run.smithi203.stdout:6 2024-03-25T19:30:00.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 10 librados: watch_flush enter 2024-03-25T19:30:00.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 10 librados: watch_flush exit 2024-03-25T19:30:00.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T19:30:00.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 10 client.4150.objecter successfully canceled tick 2024-03-25T19:30:00.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 10 monclient: shutdown 2024-03-25T19:30:00.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:00.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 >> v1:172.21.15.203:6789/0 conn(0x7f09cc142a20 legacy=0x7f09cc138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:00.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 shutdown_connections 2024-03-25T19:30:00.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 >> 172.21.15.203:0/1750862359 conn(0x7f09cc0668e0 msgr2=0x7f09cc0b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:00.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 shutdown_connections 2024-03-25T19:30:00.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.934+0000 7f09d0d39640 1 -- 172.21.15.203:0/1750862359 wait complete. 2024-03-25T19:30:00.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:00.935+0000 7f09d0d39640 1 librados: shutdown 2024-03-25T19:30:00.938 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:30:00.939+0000 7fc0f5871040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:30:00.945 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 506e8acc-eec5-4d6c-b8af-448710399c3b 7 2024-03-25T19:30:00.989 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:30:00.990+0000 7fc0f5871040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:30:01.008 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:30:01.009+0000 7f577c0c5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:30:01.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.046+0000 7f656edcf640 10 monclient: get_monmap_and_config 2024-03-25T19:30:01.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.046+0000 7f656edcf640 10 monclient: build_initial_monmap 2024-03-25T19:30:01.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.046+0000 7f656edcf640 10 monclient: monmap: 2024-03-25T19:30:01.045 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:30:01.045 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:01.046 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:30:01.046890+0000 2024-03-25T19:30:01.046 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:30:01.046890+0000 2024-03-25T19:30:01.046 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:01.046 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:01.046 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:01.046 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:01.046 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:01.046 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:01.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.046+0000 7f656edcf640 1 Processor -- start 2024-03-25T19:30:01.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.046+0000 7f656edcf640 1 -- start start 2024-03-25T19:30:01.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.046+0000 7f656edcf640 10 monclient: init 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 10 monclient(hunting): picked mon.noname-b con 0x7f65680b4700 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 10 monclient(hunting): picked mon.noname-c con 0x7f6568142a20 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 10 monclient(hunting): picked mon.noname-a con 0x7f65680a96b0 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680a9d60 con 0x7f65680a96b0 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656813ff10 con 0x7f6568142a20 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680aa1f0 con 0x7f65680b4700 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656edcf640 10 monclient(hunting): authenticate will time out at 6032.296959s 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656d36c640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f65680a96b0 0x7f656813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55582/0 (socket says 172.21.15.203:55582) 2024-03-25T19:30:01.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.047+0000 7f656d36c640 1 -- 172.21.15.203:0/3193703955 learned_addr learned my addr 172.21.15.203:0/3193703955 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:30:01.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347105725 0 0) 0x7f656813ff10 con 0x7f6568142a20 2024-03-25T19:30:01.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:01.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6558003520 con 0x7f6568142a20 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164689193 0 0) 0x7f65680aa1f0 con 0x7f65680b4700 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656813ff10 con 0x7f65680b4700 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569761572 0 0) 0x7f65680a9d60 con 0x7f65680a96b0 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:30:01.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.050+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65680aa1f0 con 0x7f65680a96b0 2024-03-25T19:30:01.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4138880756 0 0) 0x7f656813ff10 con 0x7f65680b4700 2024-03-25T19:30:01.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 >> v1:172.21.15.203:6789/0 conn(0x7f6568142a20 legacy=0x7f656813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:01.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 >> v1:172.21.15.143:6789/0 conn(0x7f65680a96b0 legacy=0x7f656813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:01.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:01.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 1 monclient: found mon.noname-b 2024-03-25T19:30:01.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:01.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680b4ae0 con 0x7f65680b4700 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f655c001670 con 0x7f65680b4700 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f65677fe640 10 monclient: dump: 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:01.054 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:01.055 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:01.055 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:01.055 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:01.055 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:01.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f656edcf640 5 monclient: authenticate success, global_id 4148 2024-03-25T19:30:01.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.053+0000 7f656edcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:01.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.055+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f655c0017d0 con 0x7f65680b4700 2024-03-25T19:30:01.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.055+0000 7f65677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:01.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.055+0000 7f65677fe640 1 -- 172.21.15.203:0/3193703955 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f655c001be0 con 0x7f65680b4700 2024-03-25T19:30:01.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.055+0000 7f65677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:01.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.055+0000 7f65677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:01.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.055+0000 7f65677fe640 10 monclient: dump: 2024-03-25T19:30:01.056 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.055+0000 7f656edcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.055+0000 7f656edcf640 4 set_mon_vals no callback set 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.056+0000 7f656edcf640 10 monclient: shutdown 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.056+0000 7f656edcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.056+0000 7f656edcf640 1 -- 172.21.15.203:0/3193703955 >> v1:172.21.15.143:6790/0 conn(0x7f65680b4700 legacy=0x7f6568142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:01.057 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.056+0000 7f656edcf640 1 -- 172.21.15.203:0/3193703955 shutdown_connections 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.056+0000 7f656edcf640 1 -- 172.21.15.203:0/3193703955 >> 172.21.15.203:0/3193703955 conn(0x7f65680668e0 msgr2=0x7f6568066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.056+0000 7f656edcf640 1 -- 172.21.15.203:0/3193703955 shutdown_connections 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.056+0000 7f656edcf640 1 -- 172.21.15.203:0/3193703955 wait complete. 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 10 monclient: build_initial_monmap 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 10 monclient: monmap: 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:01.058 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:01.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 1 librados: starting msgr at 2024-03-25T19:30:01.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 1 librados: starting objecter 2024-03-25T19:30:01.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 1 Processor -- start 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 1 -- start start 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 1 librados: setting wanted keys 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 1 librados: calling monclient init 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.057+0000 7f656edcf640 10 monclient: init 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 10 monclient(hunting): picked mon.noname-b con 0x7f65680a96b0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 10 monclient(hunting): picked mon.noname-a con 0x7f65680b4700 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 10 monclient(hunting): picked mon.noname-c con 0x7f6568142a20 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680af660 con 0x7f65680b4700 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680af940 con 0x7f65680a96b0 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680ae180 con 0x7f6568142a20 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:01.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656edcf640 10 monclient(hunting): authenticate will time out at 6032.307792s 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f6567fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f65680b4700 0x7f65680afd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.203:55586/0 (socket says 172.21.15.203:55586) 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f6567fff640 1 -- 172.21.15.203:0/3196095934 learned_addr learned my addr 172.21.15.203:0/3196095934 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.058+0000 7f656cb6b640 1 --1- 172.21.15.203:0/3196095934 >> v1:172.21.15.203:6789/0 conn(0x7f65680a96b0 0x7f65680ab880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59436/0 (socket says 172.21.15.203:59436) 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.059+0000 7f65657fa640 10 client.?.objecter ms_handle_connect 0x7f65680a96b0 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.059+0000 7f65657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.059+0000 7f65657fa640 10 client.?.objecter ms_handle_connect 0x7f65680b4700 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.059+0000 7f65657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.059+0000 7f65657fa640 10 client.?.objecter ms_handle_connect 0x7f6568142a20 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.059+0000 7f65657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640633664 0 0) 0x7f65680ae180 con 0x7f6568142a20 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:30:01.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c004eb0 con 0x7f6568142a20 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838127284 0 0) 0x7f65680af660 con 0x7f65680b4700 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65680ae180 con 0x7f65680b4700 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860589492 0 0) 0x7f65680af940 con 0x7f65680a96b0 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:30:01.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.061+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65680af660 con 0x7f65680a96b0 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.065+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1986782204 0 0) 0x7f654c004eb0 con 0x7f6568142a20 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.065+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65680af940 con 0x7f6568142a20 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.065+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6560001bd0 con 0x7f6568142a20 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.065+0000 7f65657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.065+0000 7f65657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.065+0000 7f65657fa640 10 monclient(hunting): dump: 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.065 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.066 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:01.066 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:01.066 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:01.066 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:01.066 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:01.066 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:01.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.065+0000 7f65657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:01.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.066+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103637059 0 0) 0x7f65680ae180 con 0x7f65680b4700 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.066+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f654c004eb0 con 0x7f65680b4700 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.066+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6554001950 con 0x7f65680b4700 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.066+0000 7f65657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.066+0000 7f65657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.066+0000 7f65657fa640 10 monclient(hunting): dump: 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:01.067 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.068+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1122934136 0 0) 0x7f65680af660 con 0x7f65680a96b0 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.068+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65680ae180 con 0x7f65680a96b0 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.068+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f655c001ae0 con 0x7f65680a96b0 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.068+0000 7f65657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.068+0000 7f65657fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.068+0000 7f65657fa640 10 monclient(hunting): dump: 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.068 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:01.069 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:01.069 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:01.069 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:01.069 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:01.069 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:01.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 838856355 0 0) 0x7f65680af940 con 0x7f6568142a20 2024-03-25T19:30:01.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 >> v1:172.21.15.203:6789/0 conn(0x7f65680a96b0 legacy=0x7f65680ab880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:01.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 >> v1:172.21.15.143:6789/0 conn(0x7f65680b4700 legacy=0x7f65680afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:01.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 10 monclient: _finish_hunting 0 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 1 monclient: found mon.c 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680b0b80 con 0x7f6568142a20 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 10 monclient: _finish_auth 0 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 10 monclient: _check_auth_tickets 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f65657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 5 monclient: authenticate success, global_id 4151 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 10 monclient: _renew_subs 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65680ae6a0 con 0x7f6568142a20 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 10 monclient: _renew_subs 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:01.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65680ad950 con 0x7f6568142a20 2024-03-25T19:30:01.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.069+0000 7f656edcf640 1 librados: init done 2024-03-25T19:30:01.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.070+0000 7f656edcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:01.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.070+0000 7f656edcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:01.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.070+0000 7f656edcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:01.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.070+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65680adbf0 con 0x7f6568142a20 2024-03-25T19:30:01.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.070+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6560001fe0 con 0x7f6568142a20 2024-03-25T19:30:01.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.070+0000 7f65657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:01.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.070+0000 7f6566ffd640 4 set_mon_vals no callback set 2024-03-25T19:30:01.109 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:30:01.109+0000 7fc0f5871040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6560004d00 con 0x7f6568142a20 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 10 monclient: dump: 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 3 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:30:01.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f65600052b0 con 0x7f6568142a20 2024-03-25T19:30:01.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3206113592 0 0) 0x7f6560006a70 con 0x7f6568142a20 2024-03-25T19:30:01.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 10 client.4151.objecter ms_dispatch 0x7f6568000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:30:01.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 3 client.4151.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:30:01.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 3 client.4151.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:30:01.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.167+0000 7f65657fa640 20 client.4151.objecter dump_active .. 0 homeless 2024-03-25T19:30:01.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.168+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6560006520 con 0x7f6568142a20 2024-03-25T19:30:01.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.168+0000 7f65657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:01.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.168+0000 7f65657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:01.241 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.241+0000 7f656edcf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "506e8acc-eec5-4d6c-b8af-448710399c3b", "id": 7}] 2024-03-25T19:30:01.241 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.241+0000 7f656edcf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "506e8acc-eec5-4d6c-b8af-448710399c3b", "id": 7}] 2024-03-25T19:30:01.241 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.241+0000 7f656edcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:01.241 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.241+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd new", "uuid": "506e8acc-eec5-4d6c-b8af-448710399c3b", "id": 7} v 0) v1 -- 0x7f65680ae310 con 0x7f6568142a20 2024-03-25T19:30:01.297 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:30:01.298+0000 7f577c0c5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:30:01.351 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:30:01.352+0000 7f577c0c5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:30:01.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.399+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2086127816 0 0) 0x7f6560006780 con 0x7f6568142a20 2024-03-25T19:30:01.471 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-25T19:30:01.471+0000 7f577c0c5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:30:01.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.735+0000 7f65657fa640 1 -- 172.21.15.203:0/3196095934 <== mon.2 v1:172.21.15.143:6790/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "506e8acc-eec5-4d6c-b8af-448710399c3b", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 459487092 0 1371665406) 0x7f6560015fd0 con 0x7f6568142a20 2024-03-25T19:30:01.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.735+0000 7f65657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "506e8acc-eec5-4d6c-b8af-448710399c3b", "id": 7}] 2024-03-25T19:30:01.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.735+0000 7f65657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:01.735 INFO:teuthology.orchestra.run.smithi203.stdout:7 2024-03-25T19:30:01.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 10 librados: watch_flush enter 2024-03-25T19:30:01.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 10 librados: watch_flush exit 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 20 client.4151.objecter shutdown clearing up homeless session... 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 10 client.4151.objecter successfully canceled tick 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 10 monclient: shutdown 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 >> v1:172.21.15.143:6790/0 conn(0x7f6568142a20 legacy=0x7f65680b0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 shutdown_connections 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 >> 172.21.15.203:0/3196095934 conn(0x7f65680668e0 msgr2=0x7f6568066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 shutdown_connections 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 1 -- 172.21.15.203:0/3196095934 wait complete. 2024-03-25T19:30:01.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-25T19:30:01.736+0000 7f656edcf640 1 librados: shutdown 2024-03-25T19:30:01.746 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:30:01.746 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:30:01.746 DEBUG:teuthology.orchestra.run.smithi143:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:30:01.748 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:30:01.748 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:30:01.749 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:30:01.749 DEBUG:teuthology.orchestra.run.smithi143:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:30:01.750 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:30:01.751 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:30:01.751 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:30:01.751 DEBUG:teuthology.orchestra.run.smithi143:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:30:01.753 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:30:01.753 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:30:01.753 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:30:01.753 DEBUG:teuthology.orchestra.run.smithi143:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:30:01.755 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:30:01.755 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:30:01.756 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:30:01.756 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:30:01.758 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:30:01.758 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:30:01.758 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:30:01.759 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:30:01.760 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:30:01.761 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:30:01.761 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:30:01.761 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:30:01.763 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:30:01.763 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:30:01.763 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:30:01.763 DEBUG:teuthology.orchestra.run.smithi203:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:30:01.765 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:30:01.765 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.906+0000 7f1d9d14d640 10 monclient: get_monmap_and_config 2024-03-25T19:30:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.906+0000 7f1d9d14d640 10 monclient: build_initial_monmap 2024-03-25T19:30:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.906+0000 7f1d9d14d640 10 monclient: monmap: 2024-03-25T19:30:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-03-25T19:30:01.906 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:01.907 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:30:01.907499+0000 2024-03-25T19:30:01.907 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:30:01.907499+0000 2024-03-25T19:30:01.907 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:01.907 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:01.907 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:01.907 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:01.907 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:01.907 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:30:01.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.908+0000 7f1d9d14d640 1 Processor -- start 2024-03-25T19:30:01.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.908+0000 7f1d9d14d640 1 -- start start 2024-03-25T19:30:01.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.908+0000 7f1d9d14d640 10 monclient: init 2024-03-25T19:30:01.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:01.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:01.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 10 monclient(hunting): picked mon.noname-a con 0x7f1d980b1f20 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:01.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 10 monclient(hunting): picked mon.noname-c con 0x7f1d981471a0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:01.909 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d9814ffe0 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9806e1b0 con 0x7f1d980b1f20 2024-03-25T19:30:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9806f310 con 0x7f1d981471a0 2024-03-25T19:30:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d98146c70 con 0x7f1d9814ffe0 2024-03-25T19:30:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.909+0000 7f1d9d14d640 10 monclient(hunting): authenticate will time out at 6018.252839s 2024-03-25T19:30:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.910+0000 7f1d8ffff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f1d981471a0 0x7f1d98143ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:36936/0 (socket says 172.21.15.143:36936) 2024-03-25T19:30:01.910 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.910+0000 7f1d8ffff640 1 -- 172.21.15.143:0/1206989110 learned_addr learned my addr 172.21.15.143:0/1206989110 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:01.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.937+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451631572 0 0) 0x7f1d9806e1b0 con 0x7f1d980b1f20 2024-03-25T19:30:01.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.937+0000 7f1d9659c640 10 monclient(hunting): _negotiate 2024-03-25T19:30:01.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.937+0000 7f1d9659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:01.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.937+0000 7f1d9659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:01.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.937+0000 7f1d9659c640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:30:01.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:01.937+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d84003520 con 0x7f1d980b1f20 2024-03-25T19:30:02.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480502185 0 0) 0x7f1d98146c70 con 0x7f1d9814ffe0 2024-03-25T19:30:02.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 10 monclient(hunting): _negotiate 2024-03-25T19:30:02.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:02.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:02.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:30:02.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d9806e1b0 con 0x7f1d9814ffe0 2024-03-25T19:30:02.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125937282 0 0) 0x7f1d9806f310 con 0x7f1d981471a0 2024-03-25T19:30:02.054 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 10 monclient(hunting): _negotiate 2024-03-25T19:30:02.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:02.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:02.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 10 monclient(hunting): my global_id is 4174 2024-03-25T19:30:02.055 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.054+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d98146c70 con 0x7f1d981471a0 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25158720 0 0) 0x7f1d84003520 con 0x7f1d980b1f20 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 >> v1:172.21.15.143:6790/0 conn(0x7f1d9814ffe0 legacy=0x7f1d981523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 >> v1:172.21.15.203:6789/0 conn(0x7f1d981471a0 legacy=0x7f1d98143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 10 monclient: _finish_hunting 0 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 1 monclient: found mon.noname-a 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d98147620 con 0x7f1d980b1f20 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 10 monclient: _finish_auth 0 2024-03-25T19:30:02.084 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 10 monclient: _check_auth_tickets 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9d14d640 5 monclient: authenticate success, global_id 4167 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9d14d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d88001690 con 0x7f1d980b1f20 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9659c640 10 monclient: dump: 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:02.085 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:02.086 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:02.086 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:02.086 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:30:02.086 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.084+0000 7f1d9d14d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:02.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.170+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d880018b0 con 0x7f1d980b1f20 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.170+0000 7f1d9659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.170+0000 7f1d9659c640 1 -- 172.21.15.143:0/1206989110 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d88001c40 con 0x7f1d980b1f20 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.170+0000 7f1d9d14d640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.170+0000 7f1d9d14d640 4 set_mon_vals no callback set 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.170+0000 7f1d9659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.170+0000 7f1d9659c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.170+0000 7f1d9659c640 10 monclient: dump: 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:02.170 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:02.171 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:02.171 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:02.171 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:02.171 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:30:02.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.171+0000 7f1d9d14d640 10 monclient: shutdown 2024-03-25T19:30:02.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.171+0000 7f1d9d14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:02.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.171+0000 7f1d9d14d640 1 -- 172.21.15.143:0/1206989110 >> v1:172.21.15.143:6789/0 conn(0x7f1d980b1f20 legacy=0x7f1d980b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.171+0000 7f1d9d14d640 1 -- 172.21.15.143:0/1206989110 shutdown_connections 2024-03-25T19:30:02.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.171+0000 7f1d9d14d640 1 -- 172.21.15.143:0/1206989110 >> 172.21.15.143:0/1206989110 conn(0x7f1d980b7f40 msgr2=0x7f1d980b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:02.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.171+0000 7f1d9d14d640 1 -- 172.21.15.143:0/1206989110 shutdown_connections 2024-03-25T19:30:02.172 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.171+0000 7f1d9d14d640 1 -- 172.21.15.143:0/1206989110 wait complete. 2024-03-25T19:30:02.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.172+0000 7f1d9d14d640 10 monclient: build_initial_monmap 2024-03-25T19:30:02.174 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.172+0000 7f1d9d14d640 10 monclient: monmap: 2024-03-25T19:30:02.174 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 0 2024-03-25T19:30:02.174 INFO:teuthology.orchestra.run.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:02.174 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:02.174 INFO:teuthology.orchestra.run.smithi143.stderr:created 0.000000 2024-03-25T19:30:02.174 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.173+0000 7f1d9d14d640 1 librados: starting msgr at 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.173+0000 7f1d9d14d640 1 librados: starting objecter 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.173+0000 7f1d9d14d640 1 Processor -- start 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.173+0000 7f1d9d14d640 1 -- start start 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.173+0000 7f1d9d14d640 1 librados: setting wanted keys 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.173+0000 7f1d9d14d640 1 librados: calling monclient init 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.173+0000 7f1d9d14d640 10 monclient: init 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:02.175 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:02.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 10 monclient(hunting): picked mon.noname-c con 0x7f1d980b1f20 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:02.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d981471a0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:02.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 10 monclient(hunting): picked mon.noname-a con 0x7f1d9814ffe0 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:02.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9814b040 con 0x7f1d9814ffe0 2024-03-25T19:30:02.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9814b290 con 0x7f1d981471a0 2024-03-25T19:30:02.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9814fc00 con 0x7f1d980b1f20 2024-03-25T19:30:02.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:02.176 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.174+0000 7f1d9d14d640 10 monclient(hunting): authenticate will time out at 6018.518313s 2024-03-25T19:30:02.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.175+0000 7f1d9759e640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1d9814ffe0 0x7f1d9814f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:55056/0 (socket says 172.21.15.143:55056) 2024-03-25T19:30:02.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.175+0000 7f1d9759e640 1 -- 172.21.15.143:0/3943286276 learned_addr learned my addr 172.21.15.143:0/3943286276 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:02.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.175+0000 7f1d8f7fe640 10 client.?.objecter ms_handle_connect 0x7f1d9814ffe0 2024-03-25T19:30:02.177 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.175+0000 7f1d8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:02.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.191+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534567785 0 0) 0x7f1d9814b040 con 0x7f1d9814ffe0 2024-03-25T19:30:02.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.191+0000 7f1d8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:02.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.191+0000 7f1d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:02.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.191+0000 7f1d8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:02.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.191+0000 7f1d8f7fe640 10 monclient(hunting): my global_id is 4170 2024-03-25T19:30:02.191 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.191+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d78004eb0 con 0x7f1d9814ffe0 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.204+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1682298348 0 0) 0x7f1d78004eb0 con 0x7f1d9814ffe0 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.204+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d9814b040 con 0x7f1d9814ffe0 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.204+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d90001c50 con 0x7f1d9814ffe0 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.204+0000 7f1d8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.204+0000 7f1d8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.204+0000 7f1d8f7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:02.204 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:02.205 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:02.205 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:02.205 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:02.205 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:30:02.205 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.204+0000 7f1d8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3784843979 0 0) 0x7f1d9814b040 con 0x7f1d9814ffe0 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 >> v1:172.21.15.143:6790/0 conn(0x7f1d980b1f20 legacy=0x7f1d98149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 >> v1:172.21.15.203:6789/0 conn(0x7f1d981471a0 legacy=0x7f1d9814eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 1 monclient: found mon.a 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9814f8c0 con 0x7f1d9814ffe0 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:02.209 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 5 monclient: authenticate success, global_id 4170 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 10 monclient: _renew_subs 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d98147a90 con 0x7f1d9814ffe0 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 10 monclient: _renew_subs 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9814a140 con 0x7f1d9814ffe0 2024-03-25T19:30:02.210 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.209+0000 7f1d9d14d640 1 librados: init done 2024-03-25T19:30:02.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.210+0000 7f1d9d14d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:02.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.210+0000 7f1d9d14d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:02.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.210+0000 7f1d9d14d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:02.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.210+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9804f730 con 0x7f1d9814ffe0 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.406+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d90003990 con 0x7f1d9814ffe0 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.406+0000 7f1d8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.406+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d90003d60 con 0x7f1d9814ffe0 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.406+0000 7f1d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.406+0000 7f1d8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.406+0000 7f1d8f7fe640 10 monclient: dump: 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:epoch 1 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:02.406 INFO:teuthology.orchestra.run.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:02.407 INFO:teuthology.orchestra.run.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:02.407 INFO:teuthology.orchestra.run.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:02.407 INFO:teuthology.orchestra.run.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:02.407 INFO:teuthology.orchestra.run.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:02.407 INFO:teuthology.orchestra.run.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:02.407 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:30:02.407 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.406+0000 7f1d95d9b640 4 set_mon_vals no callback set 2024-03-25T19:30:02.407 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.406+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2086127816 0 0) 0x7f1d90003f80 con 0x7f1d9814ffe0 2024-03-25T19:30:02.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.415+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f1d90015aa0 con 0x7f1d9814ffe0 2024-03-25T19:30:02.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.415+0000 7f1d8f7fe640 10 client.4170.objecter ms_dispatch 0x7f1d98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:02.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.415+0000 7f1d8f7fe640 3 client.4170.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:02.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.415+0000 7f1d8f7fe640 3 client.4170.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:02.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.415+0000 7f1d8f7fe640 20 client.4170.objecter dump_active .. 0 homeless 2024-03-25T19:30:02.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.415+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1d90013540 con 0x7f1d9814ffe0 2024-03-25T19:30:02.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.415+0000 7f1d8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:02.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.415+0000 7f1d8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:02.521 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.521+0000 7f1d9d14d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:02.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.521+0000 7f1d9d14d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:02.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.521+0000 7f1d9d14d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:02.522 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.521+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d98069f30 con 0x7f1d9814ffe0 2024-03-25T19:30:02.594 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.595+0000 7f1d8f7fe640 1 -- 172.21.15.143:0/3943286276 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f1d90013540 con 0x7f1d9814ffe0 2024-03-25T19:30:02.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.595+0000 7f1d8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:02.595 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.595+0000 7f1d8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:02.595 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-25T19:30:02.595 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:02.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 10 librados: watch_flush enter 2024-03-25T19:30:02.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 10 librados: watch_flush exit 2024-03-25T19:30:02.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 20 client.4170.objecter shutdown clearing up homeless session... 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 10 client.4170.objecter successfully canceled tick 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 10 monclient: shutdown 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 >> v1:172.21.15.143:6789/0 conn(0x7f1d9814ffe0 legacy=0x7f1d9814f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 shutdown_connections 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 >> 172.21.15.143:0/3943286276 conn(0x7f1d980b7f40 msgr2=0x7f1d98140980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.596+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 shutdown_connections 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.597+0000 7f1d9d14d640 1 -- 172.21.15.143:0/3943286276 wait complete. 2024-03-25T19:30:02.597 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-25T19:30:02.597+0000 7f1d9d14d640 1 librados: shutdown 2024-03-25T19:30:02.612 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:30:02.612 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:30:03.082 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:30:03.083+0000 7fc096ce2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:30:03.101 INFO:tasks.ceph.mgr.y.smithi143.stderr:2024-03-25T19:30:03.102+0000 7fc096ce2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:30:03.418 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.561+0000 7f43fcac8640 10 monclient: get_monmap_and_config 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.561+0000 7f43fcac8640 10 monclient: build_initial_monmap 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.561+0000 7f43fcac8640 10 monclient: monmap: 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:30:03.562341+0000 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:30:03.562341+0000 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:03.561 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:03.562 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:03.562 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.562+0000 7f43fcac8640 1 Processor -- start 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.562+0000 7f43fcac8640 1 -- start start 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.562+0000 7f43fcac8640 10 monclient: init 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.563+0000 7f43fcac8640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.563+0000 7f43fcac8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.563+0000 7f43fcac8640 10 monclient(hunting): picked mon.noname-b con 0x7f43f80b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:03.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.563+0000 7f43fcac8640 10 monclient(hunting): picked mon.noname-c con 0x7f43f80b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:03.564 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.564+0000 7f43f659c640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f43f80b4d60 0x7f43f813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:54644/0 (socket says 172.21.15.143:54644) 2024-03-25T19:30:03.564 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.564+0000 7f43fcac8640 10 monclient(hunting): picked mon.noname-a con 0x7f43f814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:03.564 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.564+0000 7f43f659c640 1 -- 172.21.15.143:0/3569269351 learned_addr learned my addr 172.21.15.143:0/3569269351 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:03.564 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.564+0000 7f43fcac8640 1 -- 172.21.15.143:0/3569269351 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f80af940 con 0x7f43f814be90 2024-03-25T19:30:03.564 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.564+0000 7f43fcac8640 1 -- 172.21.15.143:0/3569269351 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f80b0a70 con 0x7f43f80b1730 2024-03-25T19:30:03.564 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.564+0000 7f43fcac8640 1 -- 172.21.15.143:0/3569269351 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f80b0d50 con 0x7f43f80b4d60 2024-03-25T19:30:03.564 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.564+0000 7f43fcac8640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:03.564 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.564+0000 7f43fcac8640 10 monclient(hunting): authenticate will time out at 6019.907993s 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879385969 0 0) 0x7f43f80b0d50 con 0x7f43f80b4d60 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): my global_id is 4166 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e4003520 con 0x7f43f80b4d60 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605244637 0 0) 0x7f43f80b0a70 con 0x7f43f80b1730 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:03.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:30:03.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f80b0d50 con 0x7f43f80b1730 2024-03-25T19:30:03.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985848283 0 0) 0x7f43f80af940 con 0x7f43f814be90 2024-03-25T19:30:03.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:03.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:03.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:03.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 10 monclient(hunting): my global_id is 4173 2024-03-25T19:30:03.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.568+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f80b0a70 con 0x7f43f814be90 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4211610769 0 0) 0x7f43e4003520 con 0x7f43f80b4d60 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 >> v1:172.21.15.203:6789/0 conn(0x7f43f80b1730 legacy=0x7f43f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 >> v1:172.21.15.143:6789/0 conn(0x7f43f814be90 legacy=0x7f43f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 1 monclient: found mon.noname-b 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f80b1030 con 0x7f43f80b4d60 2024-03-25T19:30:03.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f43e8001690 con 0x7f43f80b4d60 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.573+0000 7f43f5d9b640 10 monclient: dump: 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:03.575 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:03.576 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:03.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.574+0000 7f43fcac8640 5 monclient: authenticate success, global_id 4166 2024-03-25T19:30:03.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.574+0000 7f43fcac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:03.577 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.575+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e80018b0 con 0x7f43f80b4d60 2024-03-25T19:30:03.577 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.575+0000 7f43f5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.575+0000 7f43fcac8640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.575+0000 7f43fcac8640 4 set_mon_vals no callback set 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43f5d9b640 1 -- 172.21.15.143:0/3569269351 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f43e8001c40 con 0x7f43f80b4d60 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43f5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43fcac8640 10 monclient: shutdown 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43f5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43f5d9b640 10 monclient: dump: 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.578 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43fcac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43fcac8640 1 -- 172.21.15.143:0/3569269351 >> v1:172.21.15.143:6790/0 conn(0x7f43f80b4d60 legacy=0x7f43f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43fcac8640 1 -- 172.21.15.143:0/3569269351 shutdown_connections 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43fcac8640 1 -- 172.21.15.143:0/3569269351 >> 172.21.15.143:0/3569269351 conn(0x7f43f80617a0 msgr2=0x7f43f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43fcac8640 1 -- 172.21.15.143:0/3569269351 shutdown_connections 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.576+0000 7f43fcac8640 1 -- 172.21.15.143:0/3569269351 wait complete. 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.577+0000 7f43fcac8640 10 monclient: build_initial_monmap 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.577+0000 7f43fcac8640 10 monclient: monmap: 2024-03-25T19:30:03.579 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.578+0000 7f43fcac8640 1 librados: starting msgr at 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.578+0000 7f43fcac8640 1 librados: starting objecter 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.578+0000 7f43fcac8640 1 Processor -- start 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.578+0000 7f43fcac8640 1 -- start start 2024-03-25T19:30:03.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.578+0000 7f43fcac8640 1 librados: setting wanted keys 2024-03-25T19:30:03.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.578+0000 7f43fcac8640 1 librados: calling monclient init 2024-03-25T19:30:03.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.578+0000 7f43fcac8640 10 monclient: init 2024-03-25T19:30:03.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:03.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 10 monclient(hunting): picked mon.noname-a con 0x7f43f80b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 10 monclient(hunting): picked mon.noname-b con 0x7f43f80b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 10 monclient(hunting): picked mon.noname-c con 0x7f43f814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f8140e50 con 0x7f43f80b1730 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f8141130 con 0x7f43f80b4d60 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f81dfd90 con 0x7f43f814be90 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.579+0000 7f43fcac8640 10 monclient(hunting): authenticate will time out at 6019.923486s 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.580+0000 7f43f6d9d640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f43f814be90 0x7f43f8142140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:54654/0 (socket says 172.21.15.143:54654) 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.580+0000 7f43f6d9d640 1 -- 172.21.15.143:0/826217773 learned_addr learned my addr 172.21.15.143:0/826217773 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.580+0000 7f43eeffd640 10 client.?.objecter ms_handle_connect 0x7f43f814be90 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.580+0000 7f43eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.580+0000 7f43eeffd640 10 client.?.objecter ms_handle_connect 0x7f43f80b4d60 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.580+0000 7f43eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:03.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.582+0000 7f43eeffd640 10 client.?.objecter ms_handle_connect 0x7f43f80b1730 2024-03-25T19:30:03.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.582+0000 7f43eeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:03.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047136870 0 0) 0x7f43f81dfd90 con 0x7f43f814be90 2024-03-25T19:30:03.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:03.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:03.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:03.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 10 monclient(hunting): my global_id is 4169 2024-03-25T19:30:03.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d8004eb0 con 0x7f43f814be90 2024-03-25T19:30:03.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917014590 0 0) 0x7f43f8141130 con 0x7f43f80b4d60 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 10 monclient(hunting): my global_id is 4180 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.584+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f81dfd90 con 0x7f43f80b4d60 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.585+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410954287 0 0) 0x7f43f8140e50 con 0x7f43f80b1730 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.585+0000 7f43eeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.585+0000 7f43eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.585+0000 7f43eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.585+0000 7f43eeffd640 10 monclient(hunting): my global_id is 4176 2024-03-25T19:30:03.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.585+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f8141130 con 0x7f43f80b1730 2024-03-25T19:30:03.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1826370492 0 0) 0x7f43f81dfd90 con 0x7f43f80b4d60 2024-03-25T19:30:03.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43f8140e50 con 0x7f43f80b4d60 2024-03-25T19:30:03.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f43e0001970 con 0x7f43f80b4d60 2024-03-25T19:30:03.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:03.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:03.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 10 monclient(hunting): dump: 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1297512066 0 0) 0x7f43d8004eb0 con 0x7f43f814be90 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.591+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43f81dfd90 con 0x7f43f814be90 2024-03-25T19:30:03.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.592+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f43f0001c90 con 0x7f43f814be90 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.592+0000 7f43eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.592+0000 7f43eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.592+0000 7f43eeffd640 10 monclient(hunting): dump: 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:03.593 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.593+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882914685 0 0) 0x7f43f8141130 con 0x7f43f80b1730 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.593+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43d8004eb0 con 0x7f43f80b1730 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.593+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f43e80015d0 con 0x7f43f80b1730 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.593+0000 7f43eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.593+0000 7f43eeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.593+0000 7f43eeffd640 10 monclient(hunting): dump: 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:03.594 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:03.595 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:03.595 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:03.595 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:03.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3027721383 0 0) 0x7f43f8140e50 con 0x7f43f80b4d60 2024-03-25T19:30:03.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 >> v1:172.21.15.143:6790/0 conn(0x7f43f814be90 legacy=0x7f43f8142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 >> v1:172.21.15.143:6789/0 conn(0x7f43f80b1730 legacy=0x7f43f81df5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:03.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:03.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 1 monclient: found mon.b 2024-03-25T19:30:03.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:03.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:03.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f813b2c0 con 0x7f43f80b4d60 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 5 monclient: authenticate success, global_id 4180 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 10 monclient: _renew_subs 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f813b590 con 0x7f43f80b4d60 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 10 monclient: _renew_subs 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f81e7800 con 0x7f43f80b4d60 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.597+0000 7f43fcac8640 1 librados: init done 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.599+0000 7f43fcac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.599+0000 7f43fcac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.599+0000 7f43fcac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:03.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.599+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f80627b0 con 0x7f43f80b4d60 2024-03-25T19:30:03.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.818+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e00035f0 con 0x7f43f80b4d60 2024-03-25T19:30:03.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.818+0000 7f43eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:03.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:03.818+0000 7f43f559a640 4 set_mon_vals no callback set 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.026+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f43e0003900 con 0x7f43f80b4d60 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.026+0000 7f43eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.026+0000 7f43eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.026+0000 7f43eeffd640 10 monclient: dump: 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:04.026 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:04.027 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.027+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 842733457 0 0) 0x7f43e0019860 con 0x7f43f80b4d60 2024-03-25T19:30:04.028 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.028+0000 7f43f659c640 10 monclient: get_auth_request con 0x7f43d806dfe0 auth_method 0 2024-03-25T19:30:04.028 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.028+0000 7f43eeffd640 10 client.4180.objecter ms_handle_connect 0x7f43d806dfe0 2024-03-25T19:30:04.030 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.031+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f43e0053570 con 0x7f43f80b4d60 2024-03-25T19:30:04.030 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.031+0000 7f43eeffd640 10 client.4180.objecter ms_dispatch 0x7f43f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.031+0000 7f43eeffd640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.031+0000 7f43eeffd640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:04.031 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.031+0000 7f43eeffd640 20 client.4180.objecter dump_active .. 0 homeless 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.034+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43e0082e60 con 0x7f43f80b4d60 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.034+0000 7f43eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:04.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.034+0000 7f43eeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:04.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.271+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f43e004eb30 con 0x7f43f80b4d60 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.400+0000 7f43fcac8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.400+0000 7f43fcac8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.400+0000 7f43fcac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:04.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.400+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43f81dff80 con 0x7f43f80b4d60 2024-03-25T19:30:04.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.472+0000 7f43eeffd640 1 -- 172.21.15.143:0/826217773 <== mon.1 v1:172.21.15.203:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f43e007a830 con 0x7f43f80b4d60 2024-03-25T19:30:04.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.472+0000 7f43eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:04.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.472+0000 7f43eeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:04.472 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:30:04.472 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:04.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.474+0000 7f43fcac8640 10 librados: watch_flush enter 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.474+0000 7f43fcac8640 10 librados: watch_flush exit 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.474+0000 7f43fcac8640 20 client.4180.objecter shutdown clearing up homeless session... 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.474+0000 7f43fcac8640 10 client.4180.objecter successfully canceled tick 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 >> v1:172.21.15.143:6800/1292282666 conn(0x7f43d806dfe0 legacy=0x7f43d80704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 10 monclient: shutdown 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 >> v1:172.21.15.203:6789/0 conn(0x7f43f80b4d60 legacy=0x7f43f8141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 shutdown_connections 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 >> 172.21.15.143:0/826217773 conn(0x7f43f80617a0 msgr2=0x7f43f813c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 shutdown_connections 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 1 -- 172.21.15.143:0/826217773 wait complete. 2024-03-25T19:30:04.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:04.475+0000 7f43fcac8640 1 librados: shutdown 2024-03-25T19:30:04.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:08.590 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:06.766 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.597 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:06.753 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.605 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:06.779 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.658 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:06.814 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.668 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:06.843 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.671 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:06.847 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.885 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:07.040 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.900 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:07.053 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.930 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:07.106 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.947 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:07.102 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.974 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:07.128 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:08.988 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:07.164 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.202 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:07.380 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.231 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-25T19:30:09.222+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:30:09.247 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-25T19:30:09.241+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:30:09.261 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:07.440 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.287 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-25T19:30:09.278+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:30:09.338 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:07.494 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.396 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:07.552 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.493 INFO:tasks.ceph.osd.2.smithi143.stderr:2024-03-25T19:30:09.488+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:30:09.551 INFO:tasks.ceph.osd.1.smithi143.stderr:2024-03-25T19:30:09.545+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:30:09.576 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-25T19:30:09.567+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:30:09.705 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:07.861 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.771 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:07.947 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.782 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:07.956 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.829 INFO:tasks.ceph.osd.0.smithi143.stderr:2024-03-25T19:30:09.824+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:30:09.960 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:08.115 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:09.969 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-03-25T19:30:09.963+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:30:10.006 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:08.160 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.046 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:08.222 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.079 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:08.235 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.139 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:08.316 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.154 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:08.329 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.290 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:08.468 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.336 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:08.491 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.382 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:08.536 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.411 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:08.587 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.434 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:08.591 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.440 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:08.596 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.501 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:08.677 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.525 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:08.700 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.652 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:08.831 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.694 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:08.849 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.700 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:08.854 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.732 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:08.889 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.776 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:08.952 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.792 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:08.967 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.793 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:08.969 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.798 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:08.955 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.965 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:09.119 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.967 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:09.145 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:10.989 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:09.144 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.020 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.176 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.048 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.224 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.086 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.261 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.090 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.266 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.115 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:09.272 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.232 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:09.386 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.240 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:09.418 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.276 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:09.431 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.293 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:11.308 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.465 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.337 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.514 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.382+0000 7fbe2759e640 10 monclient: get_monmap_and_config 2024-03-25T19:30:11.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.382+0000 7fbe2759e640 10 monclient: build_initial_monmap 2024-03-25T19:30:11.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.382+0000 7fbe2759e640 10 monclient: monmap: 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:30:11.383071+0000 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:30:11.383071+0000 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:11.382 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.382+0000 7fbe2759e640 1 Processor -- start 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.382+0000 7fbe2759e640 1 -- start start 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.382+0000 7fbe2759e640 10 monclient: init 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 10 monclient(hunting): picked mon.noname-b con 0x7fbe280b49a0 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 10 monclient(hunting): picked mon.noname-a con 0x7fbe280a9680 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 10 monclient(hunting): picked mon.noname-c con 0x7fbe280b3fa0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:11.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2813fe70 con 0x7fbe280a9680 2024-03-25T19:30:11.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe280ab250 con 0x7fbe280b3fa0 2024-03-25T19:30:11.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe280a9b10 con 0x7fbe280b49a0 2024-03-25T19:30:11.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:11.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe2759e640 10 monclient(hunting): authenticate will time out at 6027.727172s 2024-03-25T19:30:11.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe25d9b640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fbe280a9680 0x7fbe28145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:33694/0 (socket says 172.21.15.143:33694) 2024-03-25T19:30:11.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.383+0000 7fbe25d9b640 1 -- 172.21.15.143:0/1367521587 learned_addr learned my addr 172.21.15.143:0/1367521587 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:11.385 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.559 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067817576 0 0) 0x7fbe1c001490 con 0x7fbe280a9680 2024-03-25T19:30:11.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): my global_id is 4203 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe10003520 con 0x7fbe280a9680 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291042218 0 0) 0x7fbe280a9b10 con 0x7fbe280b49a0 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): my global_id is 4196 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1c001490 con 0x7fbe280b49a0 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565719002 0 0) 0x7fbe280ab250 con 0x7fbe280b3fa0 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:30:11.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.386+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe280a9b10 con 0x7fbe280b3fa0 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2259570887 0 0) 0x7fbe1c001490 con 0x7fbe280b49a0 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 >> v1:172.21.15.203:6789/0 conn(0x7fbe280b3fa0 legacy=0x7fbe28145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 >> v1:172.21.15.143:6789/0 conn(0x7fbe280a9680 legacy=0x7fbe28145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 10 monclient: _finish_hunting 0 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 1 monclient: found mon.noname-b 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe28151b50 con 0x7fbe280b49a0 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 10 monclient: _finish_auth 0 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 10 monclient: _check_auth_tickets 2024-03-25T19:30:11.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbe20001670 con 0x7fbe280b49a0 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2759e640 5 monclient: authenticate success, global_id 4196 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2559a640 10 monclient: dump: 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:11.391 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:11.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.389+0000 7fbe2759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:11.392 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.566 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.391+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe200017d0 con 0x7fbe280b49a0 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.391+0000 7fbe2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.391+0000 7fbe2559a640 1 -- 172.21.15.143:0/1367521587 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbe20001ba0 con 0x7fbe280b49a0 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.391+0000 7fbe2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.391+0000 7fbe2559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.391+0000 7fbe2559a640 10 monclient: dump: 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.393 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:11.394 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:11.394 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:11.394 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:11.394 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:11.394 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.391+0000 7fbe2759e640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:11.394 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.391+0000 7fbe2759e640 4 set_mon_vals no callback set 2024-03-25T19:30:11.394 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:09.548 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.392+0000 7fbe2759e640 10 monclient: shutdown 2024-03-25T19:30:11.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.392+0000 7fbe2759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:11.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.392+0000 7fbe2759e640 1 -- 172.21.15.143:0/1367521587 >> v1:172.21.15.143:6790/0 conn(0x7fbe280b49a0 legacy=0x7fbe2813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.392+0000 7fbe2759e640 1 -- 172.21.15.143:0/1367521587 shutdown_connections 2024-03-25T19:30:11.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.392+0000 7fbe2759e640 1 -- 172.21.15.143:0/1367521587 >> 172.21.15.143:0/1367521587 conn(0x7fbe280615c0 msgr2=0x7fbe280619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:11.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.392+0000 7fbe2759e640 1 -- 172.21.15.143:0/1367521587 shutdown_connections 2024-03-25T19:30:11.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.392+0000 7fbe2759e640 1 -- 172.21.15.143:0/1367521587 wait complete. 2024-03-25T19:30:11.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 10 monclient: build_initial_monmap 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 10 monclient: monmap: 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 1 librados: starting msgr at 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 1 librados: starting objecter 2024-03-25T19:30:11.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 1 Processor -- start 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 1 -- start start 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 1 librados: setting wanted keys 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 1 librados: calling monclient init 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.393+0000 7fbe2759e640 10 monclient: init 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 10 monclient(hunting): picked mon.noname-b con 0x7fbe280b3fa0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 10 monclient(hunting): picked mon.noname-c con 0x7fbe280b49a0 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 10 monclient(hunting): picked mon.noname-a con 0x7fbe2814fa60 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2812b770 con 0x7fbe2814fa60 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe28144ab0 con 0x7fbe280b3fa0 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe28144d90 con 0x7fbe280b49a0 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:11.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe2759e640 10 monclient(hunting): authenticate will time out at 6027.737853s 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe25d9b640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fbe280b49a0 0x7fbe2812a950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:48284/0 (socket says 172.21.15.143:48284) 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe25d9b640 1 -- 172.21.15.143:0/3908777215 learned_addr learned my addr 172.21.15.143:0/3908777215 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe16ffd640 10 client.?.objecter ms_handle_connect 0x7fbe280b49a0 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.394+0000 7fbe16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.395+0000 7fbe16ffd640 10 client.?.objecter ms_handle_connect 0x7fbe2814fa60 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.395+0000 7fbe16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.395+0000 7fbe16ffd640 10 client.?.objecter ms_handle_connect 0x7fbe280b3fa0 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.395+0000 7fbe16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811506896 0 0) 0x7fbe28144ab0 con 0x7fbe280b3fa0 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe08004eb0 con 0x7fbe280b3fa0 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773610062 0 0) 0x7fbe2812b770 con 0x7fbe2814fa60 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe28144ab0 con 0x7fbe2814fa60 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256278722 0 0) 0x7fbe28144d90 con 0x7fbe280b49a0 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:11.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.396+0000 7fbe16ffd640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:30:11.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.397+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe2812b770 con 0x7fbe280b49a0 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 188125908 0 0) 0x7fbe2812b770 con 0x7fbe280b49a0 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe28144d90 con 0x7fbe280b49a0 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbe1c001710 con 0x7fbe280b49a0 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.403 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1326572645 0 0) 0x7fbe08004eb0 con 0x7fbe280b3fa0 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe2812b770 con 0x7fbe280b3fa0 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbe20001ae0 con 0x7fbe280b3fa0 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:11.404 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3078643219 0 0) 0x7fbe28144ab0 con 0x7fbe2814fa60 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe08004eb0 con 0x7fbe2814fa60 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbe18001870 con 0x7fbe2814fa60 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:11.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.401+0000 7fbe16ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:11.406 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:11.407 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.404+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3454676416 0 0) 0x7fbe28144d90 con 0x7fbe280b49a0 2024-03-25T19:30:11.407 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.404+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 >> v1:172.21.15.203:6789/0 conn(0x7fbe280b3fa0 legacy=0x7fbe2814f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.404+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 >> v1:172.21.15.143:6789/0 conn(0x7fbe2814fa60 legacy=0x7fbe2812b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe16ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe16ffd640 1 monclient: found mon.c 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe28143340 con 0x7fbe280b49a0 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe16ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe16ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 5 monclient: authenticate success, global_id 4199 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 10 monclient: _renew_subs 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe28143620 con 0x7fbe280b49a0 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 10 monclient: _renew_subs 2024-03-25T19:30:11.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:11.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2813e5b0 con 0x7fbe280b49a0 2024-03-25T19:30:11.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 1 librados: init done 2024-03-25T19:30:11.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.405+0000 7fbe2759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.406+0000 7fbe2759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.406+0000 7fbe2759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:11.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.406+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe280627b0 con 0x7fbe280b49a0 2024-03-25T19:30:11.497 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:09.651 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.509 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:09.687 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.564 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:09.719 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.596 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:09.753 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.620+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe1c004b60 con 0x7fbe280b49a0 2024-03-25T19:30:11.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.620+0000 7fbe16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:11.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.620+0000 7fbe24d99640 4 set_mon_vals no callback set 2024-03-25T19:30:11.636 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:09.813 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.648+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbe1c004e70 con 0x7fbe280b49a0 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.648+0000 7fbe16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.648+0000 7fbe16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.648+0000 7fbe16ffd640 10 monclient: dump: 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:11.648 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:11.649 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:11.649 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:11.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.649+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fbe1c01ae40 con 0x7fbe280b49a0 2024-03-25T19:30:11.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.649+0000 7fbe2659c640 10 monclient: get_auth_request con 0x7fbe0806e050 auth_method 0 2024-03-25T19:30:11.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.650+0000 7fbe16ffd640 10 client.4199.objecter ms_handle_connect 0x7fbe0806e050 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.655+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fbe1c054ba0 con 0x7fbe280b49a0 2024-03-25T19:30:11.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.655+0000 7fbe16ffd640 10 client.4199.objecter ms_dispatch 0x7fbe28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:11.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.655+0000 7fbe16ffd640 3 client.4199.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:11.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.655+0000 7fbe16ffd640 3 client.4199.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:11.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.655+0000 7fbe16ffd640 20 client.4199.objecter dump_active .. 0 homeless 2024-03-25T19:30:11.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.655+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe1c0504e0 con 0x7fbe280b49a0 2024-03-25T19:30:11.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.655+0000 7fbe16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:11.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.655+0000 7fbe16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:11.657 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:09.832 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.658 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:09.815 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.685 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:09.861 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.761 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:09.915 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.783 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:09.961 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.830+0000 7fbe2759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:11.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.830+0000 7fbe2759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:11.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.830+0000 7fbe2759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:30:11.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.830+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe2814e900 con 0x7fbe280b49a0 2024-03-25T19:30:11.859 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:10.014 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.876+0000 7fbe16ffd640 1 -- 172.21.15.143:0/3908777215 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fbe1c0525e0 con 0x7fbe280b49a0 2024-03-25T19:30:11.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.876+0000 7fbe16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:11.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.876+0000 7fbe16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:11.877 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:30:11.877 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.878+0000 7fbe2759e640 10 librados: watch_flush enter 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.878+0000 7fbe2759e640 10 librados: watch_flush exit 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.878+0000 7fbe2759e640 20 client.4199.objecter shutdown clearing up homeless session... 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.878+0000 7fbe2759e640 10 client.4199.objecter successfully canceled tick 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.878+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 >> v1:172.21.15.143:6800/1292282666 conn(0x7fbe0806e050 legacy=0x7fbe08070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.879+0000 7fbe2759e640 10 monclient: shutdown 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.879+0000 7fbe2759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.879+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 >> v1:172.21.15.143:6790/0 conn(0x7fbe280b49a0 legacy=0x7fbe2812a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.879+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 shutdown_connections 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.879+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 >> 172.21.15.143:0/3908777215 conn(0x7fbe280615c0 msgr2=0x7fbe280b71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.879+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 shutdown_connections 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.879+0000 7fbe2759e640 1 -- 172.21.15.143:0/3908777215 wait complete. 2024-03-25T19:30:11.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:11.879+0000 7fbe2759e640 1 librados: shutdown 2024-03-25T19:30:11.886 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.042 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:11.926 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:10.083 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.930 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.106 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.955 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.130 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.975 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.151 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:11.980 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.156 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.031 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:10.185 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.049 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:10.227 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.152 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:10.307 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.176 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.333 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.194 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:10.351 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.243 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.418 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.269 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:10.445 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.278 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:10.454 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.295 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:10.449 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.326 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:10.504 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.443 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:10.598 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.465 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:10.621 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.591 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:10.769 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.702 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.858 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.732 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:10.889 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.762 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:10.916 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.789 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:10.964 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.794 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:10.951 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.819 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:10.973 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.835 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:11.011 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.851 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:11.027 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.884 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:11.059 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.893 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:11.069 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.941 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:11.117 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:12.982 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:11.137 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:13.036 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:11.191 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:13.061 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:11.239 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:13.138 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:11.316 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:13.209 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:11.366 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:13.263 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:11.420 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:18.370 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:16.526 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:18.468 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:16.644 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:18.479 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:16.633 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:18.489 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:16.663 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:18.503 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:16.680 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:18.696 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:18.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.786+0000 7f599b7c9640 10 monclient: get_monmap_and_config 2024-03-25T19:30:18.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.786+0000 7f599b7c9640 10 monclient: build_initial_monmap 2024-03-25T19:30:18.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.786+0000 7f599b7c9640 10 monclient: monmap: 2024-03-25T19:30:18.785 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:30:18.786939+0000 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:30:18.786939+0000 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:18.786 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:18.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.786+0000 7f599b7c9640 1 Processor -- start 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.786+0000 7f599b7c9640 1 -- start start 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.786+0000 7f599b7c9640 10 monclient: init 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 10 monclient(hunting): picked mon.noname-c con 0x7f59940b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 10 monclient(hunting): picked mon.noname-b con 0x7f59940b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 10 monclient(hunting): picked mon.noname-a con 0x7f599414be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59940af940 con 0x7f599414be90 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59940b0a70 con 0x7f59940b4d60 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59940b0d50 con 0x7f59940b1730 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.787+0000 7f599b7c9640 10 monclient(hunting): authenticate will time out at 6035.131169s 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.788+0000 7f5999d66640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f599414be90 0x7f599414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:59582/0 (socket says 172.21.15.143:59582) 2024-03-25T19:30:18.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.788+0000 7f5999d66640 1 -- 172.21.15.143:0/2453529323 learned_addr learned my addr 172.21.15.143:0/2453529323 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:18.790 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.790+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897615333 0 0) 0x7f59940b0a70 con 0x7f59940b4d60 2024-03-25T19:30:18.790 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.790+0000 7f598bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.790 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.790+0000 7f598bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.790 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.790+0000 7f598bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.790 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.790+0000 7f598bfff640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:30:18.790 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.790+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5980003520 con 0x7f59940b4d60 2024-03-25T19:30:18.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147915441 0 0) 0x7f59940b0d50 con 0x7f59940b1730 2024-03-25T19:30:18.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:30:18.792 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59940b0a70 con 0x7f59940b1730 2024-03-25T19:30:18.792 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441480287 0 0) 0x7f59940af940 con 0x7f599414be90 2024-03-25T19:30:18.792 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.792 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.792 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.792 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:30:18.792 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.791+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59940b0d50 con 0x7f599414be90 2024-03-25T19:30:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2034206299 0 0) 0x7f5980003520 con 0x7f59940b4d60 2024-03-25T19:30:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 >> v1:172.21.15.143:6790/0 conn(0x7f59940b1730 legacy=0x7f59940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 >> v1:172.21.15.143:6789/0 conn(0x7f599414be90 legacy=0x7f599414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 1 monclient: found mon.noname-c 2024-03-25T19:30:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59940b1030 con 0x7f59940b4d60 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 10 monclient: _finish_auth 0 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f59900015c0 con 0x7f59940b4d60 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f598bfff640 10 monclient: dump: 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:18.795 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:18.796 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:18.796 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:18.796 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:18.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f599b7c9640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:30:18.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.793+0000 7f599b7c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:18.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.796+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5990001720 con 0x7f59940b4d60 2024-03-25T19:30:18.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.796+0000 7f598bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:18.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.796+0000 7f598bfff640 1 -- 172.21.15.143:0/2453529323 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5990001b30 con 0x7f59940b4d60 2024-03-25T19:30:18.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.796+0000 7f598bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.796+0000 7f598bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:18.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.796+0000 7f598bfff640 10 monclient: dump: 2024-03-25T19:30:18.798 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:18.798 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.796+0000 7f599b7c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.796+0000 7f599b7c9640 4 set_mon_vals no callback set 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.797+0000 7f599b7c9640 10 monclient: shutdown 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.797+0000 7f599b7c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.797+0000 7f599b7c9640 1 -- 172.21.15.143:0/2453529323 >> v1:172.21.15.203:6789/0 conn(0x7f59940b4d60 legacy=0x7f599413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.797+0000 7f599b7c9640 1 -- 172.21.15.143:0/2453529323 shutdown_connections 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.797+0000 7f599b7c9640 1 -- 172.21.15.143:0/2453529323 >> 172.21.15.143:0/2453529323 conn(0x7f59940617a0 msgr2=0x7f5994061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.797+0000 7f599b7c9640 1 -- 172.21.15.143:0/2453529323 shutdown_connections 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.797+0000 7f599b7c9640 1 -- 172.21.15.143:0/2453529323 wait complete. 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 10 monclient: build_initial_monmap 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 10 monclient: monmap: 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:18.800 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:18.801 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:18.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 1 librados: starting msgr at 2024-03-25T19:30:18.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 1 librados: starting objecter 2024-03-25T19:30:18.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 1 Processor -- start 2024-03-25T19:30:18.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 1 -- start start 2024-03-25T19:30:18.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 1 librados: setting wanted keys 2024-03-25T19:30:18.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 1 librados: calling monclient init 2024-03-25T19:30:18.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.798+0000 7f599b7c9640 10 monclient: init 2024-03-25T19:30:18.804 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:18.804 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:18.804 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 10 monclient(hunting): picked mon.noname-b con 0x7f59940b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:18.804 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 10 monclient(hunting): picked mon.noname-a con 0x7f59940b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 10 monclient(hunting): picked mon.noname-c con 0x7f599414be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59941418e0 con 0x7f59940b4d60 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5994141b50 con 0x7f59940b1730 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5994141e30 con 0x7f599414be90 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.799+0000 7f599b7c9640 10 monclient(hunting): authenticate will time out at 6035.143398s 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.800+0000 7f5999d66640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f599414be90 0x7f59941e6a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:54368/0 (socket says 172.21.15.143:54368) 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.800+0000 7f5999d66640 1 -- 172.21.15.143:0/89984370 learned_addr learned my addr 172.21.15.143:0/89984370 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.800+0000 7f5989ffb640 10 client.?.objecter ms_handle_connect 0x7f59940b1730 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.800+0000 7f5989ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.800+0000 7f5989ffb640 10 client.?.objecter ms_handle_connect 0x7f599414be90 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.800+0000 7f5989ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.800+0000 7f5989ffb640 10 client.?.objecter ms_handle_connect 0x7f59940b4d60 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.800+0000 7f5989ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:18.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867084896 0 0) 0x7f5994141e30 con 0x7f599414be90 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5978004eb0 con 0x7f599414be90 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733195507 0 0) 0x7f5994141b50 con 0x7f59940b1730 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.802+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5994141e30 con 0x7f59940b1730 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.803+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794035370 0 0) 0x7f59941418e0 con 0x7f59940b4d60 2024-03-25T19:30:18.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.803+0000 7f5989ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:18.807 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.803+0000 7f5989ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:18.807 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.803+0000 7f5989ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:18.807 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.803+0000 7f5989ffb640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:30:18.807 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.803+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5994141b50 con 0x7f59940b4d60 2024-03-25T19:30:18.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.806+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 389499866 0 0) 0x7f5994141e30 con 0x7f59940b1730 2024-03-25T19:30:18.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.806+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59941418e0 con 0x7f59940b1730 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.806+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5990001a30 con 0x7f59940b1730 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.807+0000 7f5989ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.807+0000 7f5989ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.807+0000 7f5989ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:18.809 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.807+0000 7f5989ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.807+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2159656669 0 0) 0x7f5978004eb0 con 0x7f599414be90 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.807+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5994141e30 con 0x7f599414be90 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.808+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f598c001bb0 con 0x7f599414be90 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.808+0000 7f5989ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.808+0000 7f5989ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.808+0000 7f5989ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:18.810 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:18.811 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:18.811 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:18.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 176563813 0 0) 0x7f59941418e0 con 0x7f59940b1730 2024-03-25T19:30:18.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 >> v1:172.21.15.143:6790/0 conn(0x7f599414be90 legacy=0x7f59941e6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 >> v1:172.21.15.143:6789/0 conn(0x7f59940b4d60 legacy=0x7f5994135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:18.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:18.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 1 monclient: found mon.b 2024-03-25T19:30:18.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:18.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:18.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5994138080 con 0x7f59940b1730 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.810+0000 7f5989ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 10 monclient: _renew_subs 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5994138a20 con 0x7f59940b1730 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 10 monclient: _renew_subs 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5994136320 con 0x7f59940b1730 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.811+0000 7f599b7c9640 1 librados: init done 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.812+0000 7f599b7c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.812+0000 7f599b7c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:18.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.812+0000 7f599b7c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:18.815 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.812+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59941401c0 con 0x7f59940b1730 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.829+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59900038a0 con 0x7f59940b1730 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.829+0000 7f5989ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.829+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5990003bb0 con 0x7f59940b1730 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.829+0000 7f5989ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.829+0000 7f5989ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.829+0000 7f5989ffb640 10 monclient: dump: 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:18.829 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:18.830 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:18.830 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:18.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.829+0000 7f598b7fe640 4 set_mon_vals no callback set 2024-03-25T19:30:18.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.830+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f5990019b80 con 0x7f59940b1730 2024-03-25T19:30:18.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.831+0000 7f5998d64640 10 monclient: get_auth_request con 0x7f597806df30 auth_method 0 2024-03-25T19:30:18.831 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:16.986 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:18.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.831+0000 7f5989ffb640 10 client.4216.objecter ms_handle_connect 0x7f597806df30 2024-03-25T19:30:18.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.832+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f59900535d0 con 0x7f59940b1730 2024-03-25T19:30:18.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.832+0000 7f5989ffb640 10 client.4216.objecter ms_dispatch 0x7f5994000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:18.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.832+0000 7f5989ffb640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:18.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.832+0000 7f5989ffb640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:18.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:18.832+0000 7f5989ffb640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T19:30:18.923 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:17.102 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:19.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.073+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f599004f0e0 con 0x7f59940b1730 2024-03-25T19:30:19.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.073+0000 7f5989ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:19.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.073+0000 7f5989ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:19.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.299+0000 7f599b7c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:19.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.299+0000 7f599b7c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:19.299 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.299+0000 7f599b7c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:19.299 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.299+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5994138210 con 0x7f59940b1730 2024-03-25T19:30:19.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.325+0000 7f5989ffb640 1 -- 172.21.15.143:0/89984370 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f5994138210 con 0x7f59940b1730 2024-03-25T19:30:19.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.325+0000 7f5989ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:19.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.325+0000 7f5989ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:19.325 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:30:19.325 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:19.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 10 librados: watch_flush enter 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 10 librados: watch_flush exit 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 10 client.4216.objecter successfully canceled tick 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 >> v1:172.21.15.143:6800/1292282666 conn(0x7f597806df30 legacy=0x7f59780703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 10 monclient: shutdown 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 >> v1:172.21.15.203:6789/0 conn(0x7f59940b1730 legacy=0x7f59941405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 shutdown_connections 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 >> 172.21.15.143:0/89984370 conn(0x7f59940617a0 msgr2=0x7f59940b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 shutdown_connections 2024-03-25T19:30:19.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 1 -- 172.21.15.143:0/89984370 wait complete. 2024-03-25T19:30:19.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:19.328+0000 7f599b7c9640 1 librados: shutdown 2024-03-25T19:30:19.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:19.368 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:17.525 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:23.695 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:21.851 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:23.698 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:21.855 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:23.915 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:22.091 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:23.919 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:22.095 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:23.942 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:22.118 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:23.945 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:22.122 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:23.994 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:22.168 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:23.998 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:22.172 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:24.382 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:22.535 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:24.386 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:22.539 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:24.590 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:22.745 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:24.597 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:22.752 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:25.094 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:23.272 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:25.097 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:23.276 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:25.432 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:23.588 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:25.437 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:23.594 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:26.144 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:26.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.234+0000 7f415d91c640 10 monclient: get_monmap_and_config 2024-03-25T19:30:26.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.234+0000 7f415d91c640 10 monclient: build_initial_monmap 2024-03-25T19:30:26.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.234+0000 7f415d91c640 10 monclient: monmap: 2024-03-25T19:30:26.234 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:26.234 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:26.234 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:30:26.235557+0000 2024-03-25T19:30:26.235 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:30:26.235557+0000 2024-03-25T19:30:26.235 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:26.235 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:26.235 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:26.235 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:26.235 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:26.235 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:26.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 1 Processor -- start 2024-03-25T19:30:26.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 1 -- start start 2024-03-25T19:30:26.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 10 monclient: init 2024-03-25T19:30:26.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 10 monclient(hunting): picked mon.noname-b con 0x7f41580b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 10 monclient(hunting): picked mon.noname-c con 0x7f41580b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 10 monclient(hunting): picked mon.noname-a con 0x7f415814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580af940 con 0x7f415814be90 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580b0a70 con 0x7f41580b1730 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41580b0d50 con 0x7f41580b4d60 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.235+0000 7f415d91c640 10 monclient(hunting): authenticate will time out at 6042.579629s 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.236+0000 7f4156ffd640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f41580b4d60 0x7f415813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:54382/0 (socket says 172.21.15.143:54382) 2024-03-25T19:30:26.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.236+0000 7f4156ffd640 1 -- 172.21.15.143:0/1575027341 learned_addr learned my addr 172.21.15.143:0/1575027341 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:26.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.239+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307694771 0 0) 0x7f41580b0a70 con 0x7f41580b1730 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140003520 con 0x7f41580b1730 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375137292 0 0) 0x7f41580b0d50 con 0x7f41580b4d60 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:30:26.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41580b0a70 con 0x7f41580b4d60 2024-03-25T19:30:26.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684330114 0 0) 0x7f41580af940 con 0x7f415814be90 2024-03-25T19:30:26.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:30:26.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.240+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41580b0d50 con 0x7f415814be90 2024-03-25T19:30:26.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2972405871 0 0) 0x7f4140003520 con 0x7f41580b1730 2024-03-25T19:30:26.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 >> v1:172.21.15.143:6790/0 conn(0x7f41580b4d60 legacy=0x7f415813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 >> v1:172.21.15.143:6789/0 conn(0x7f415814be90 legacy=0x7f415814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:26.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 1 monclient: found mon.noname-c 2024-03-25T19:30:26.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580b1030 con 0x7f41580b1730 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f414c001690 con 0x7f41580b1730 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f4155ffb640 10 monclient: dump: 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:26.245 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f415d91c640 5 monclient: authenticate success, global_id 4219 2024-03-25T19:30:26.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.243+0000 7f415d91c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:26.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.245+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f414c0018e0 con 0x7f41580b1730 2024-03-25T19:30:26.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.245+0000 7f4155ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:26.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.245+0000 7f415d91c640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:26.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.245+0000 7f415d91c640 4 set_mon_vals no callback set 2024-03-25T19:30:26.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f4155ffb640 1 -- 172.21.15.143:0/1575027341 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f414c001c70 con 0x7f41580b1730 2024-03-25T19:30:26.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f4155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f4155ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f4155ffb640 10 monclient: dump: 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:26.249 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f415d91c640 10 monclient: shutdown 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f415d91c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f415d91c640 1 -- 172.21.15.143:0/1575027341 >> v1:172.21.15.203:6789/0 conn(0x7f41580b1730 legacy=0x7f41580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f415d91c640 1 -- 172.21.15.143:0/1575027341 shutdown_connections 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f415d91c640 1 -- 172.21.15.143:0/1575027341 >> 172.21.15.143:0/1575027341 conn(0x7f41580617a0 msgr2=0x7f4158061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f415d91c640 1 -- 172.21.15.143:0/1575027341 shutdown_connections 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.246+0000 7f415d91c640 1 -- 172.21.15.143:0/1575027341 wait complete. 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.247+0000 7f415d91c640 10 monclient: build_initial_monmap 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.247+0000 7f415d91c640 10 monclient: monmap: 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:26.250 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.248+0000 7f415d91c640 1 librados: starting msgr at 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.248+0000 7f415d91c640 1 librados: starting objecter 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.248+0000 7f415d91c640 1 Processor -- start 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.248+0000 7f415d91c640 1 -- start start 2024-03-25T19:30:26.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.248+0000 7f415d91c640 1 librados: setting wanted keys 2024-03-25T19:30:26.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.248+0000 7f415d91c640 1 librados: calling monclient init 2024-03-25T19:30:26.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.248+0000 7f415d91c640 10 monclient: init 2024-03-25T19:30:26.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.249+0000 7f415d91c640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:26.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.249+0000 7f415d91c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:26.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.249+0000 7f415d91c640 10 monclient(hunting): picked mon.noname-a con 0x7f41580b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:26.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.249+0000 7f415d91c640 10 monclient(hunting): picked mon.noname-b con 0x7f41580b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:26.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f4156ffd640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f41580b1730 0x7f4158134d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:59602/0 (socket says 172.21.15.143:59602) 2024-03-25T19:30:26.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f4156ffd640 1 -- 172.21.15.143:0/419780447 learned_addr learned my addr 172.21.15.143:0/419780447 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:26.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f415d91c640 10 monclient(hunting): picked mon.noname-c con 0x7f415814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:26.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41581406b0 con 0x7f41580b1730 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4158140380 con 0x7f41580b4d60 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415813b300 con 0x7f415814be90 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f415d91c640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f415d91c640 10 monclient(hunting): authenticate will time out at 6042.593867s 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f413ffff640 10 client.?.objecter ms_handle_connect 0x7f41580b4d60 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f413ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f413ffff640 10 client.?.objecter ms_handle_connect 0x7f41580b1730 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.250+0000 7f413ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.251+0000 7f413ffff640 10 client.?.objecter ms_handle_connect 0x7f415814be90 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.251+0000 7f413ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715710561 0 0) 0x7f4158140380 con 0x7f41580b4d60 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4144004eb0 con 0x7f41580b4d60 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145156291 0 0) 0x7f41581406b0 con 0x7f41580b1730 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:30:26.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.253+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4158140380 con 0x7f41580b1730 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.256+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921062224 0 0) 0x7f415813b300 con 0x7f415814be90 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.256+0000 7f413ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.256+0000 7f413ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.256+0000 7f413ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.256+0000 7f413ffff640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.256+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41581406b0 con 0x7f415814be90 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.257+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 725723093 0 0) 0x7f4158140380 con 0x7f41580b1730 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.257+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415813b300 con 0x7f41580b1730 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.257+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4148001930 con 0x7f41580b1730 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.257+0000 7f413ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.257+0000 7f413ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.257+0000 7f413ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:26.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.257+0000 7f413ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.259+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2314431139 0 0) 0x7f4144004eb0 con 0x7f41580b4d60 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.259+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4158140380 con 0x7f41580b4d60 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.259+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f414c001bf0 con 0x7f41580b4d60 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.259+0000 7f413ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.259+0000 7f413ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.259+0000 7f413ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:26.262 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:26.263 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:26.263 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:26.263 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:26.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2178382580 0 0) 0x7f415813b300 con 0x7f41580b1730 2024-03-25T19:30:26.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 >> v1:172.21.15.143:6790/0 conn(0x7f415814be90 legacy=0x7f41581db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 >> v1:172.21.15.203:6789/0 conn(0x7f41580b4d60 legacy=0x7f41581dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:26.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 1 monclient: found mon.a 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415813b490 con 0x7f41580b1730 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 10 monclient: _finish_auth 0 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f413ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 5 monclient: authenticate success, global_id 4218 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 10 monclient: _renew_subs 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415813a860 con 0x7f41580b1730 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 10 monclient: _renew_subs 2024-03-25T19:30:26.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:26.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4158144bc0 con 0x7f41580b1730 2024-03-25T19:30:26.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.261+0000 7f415d91c640 1 librados: init done 2024-03-25T19:30:26.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.262+0000 7f415d91c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:26.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.262+0000 7f415d91c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:26.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.262+0000 7f415d91c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:26.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.262+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41580627b0 con 0x7f41580b1730 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.623+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4148001d60 con 0x7f41580b1730 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.623+0000 7f413ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.623+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4148004980 con 0x7f41580b1730 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.623+0000 7f413ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.623+0000 7f413ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.623+0000 7f413ffff640 10 monclient: dump: 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:26.623 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.624 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:26.624 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:26.624 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:26.624 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:26.624 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:26.624 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:26.624 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:26.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.623+0000 7f41557fa640 4 set_mon_vals no callback set 2024-03-25T19:30:26.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.624+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f414801a9d0 con 0x7f41580b1730 2024-03-25T19:30:26.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.625+0000 7f41567fc640 10 monclient: get_auth_request con 0x7f414406df20 auth_method 0 2024-03-25T19:30:26.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.625+0000 7f413ffff640 10 client.4218.objecter ms_handle_connect 0x7f414406df20 2024-03-25T19:30:26.632 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.633+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f41480547a0 con 0x7f41580b1730 2024-03-25T19:30:26.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.633+0000 7f413ffff640 10 client.4218.objecter ms_dispatch 0x7f4158000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:26.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.633+0000 7f413ffff640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:26.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.633+0000 7f413ffff640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:26.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.633+0000 7f413ffff640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T19:30:26.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.633+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41480500e0 con 0x7f41580b1730 2024-03-25T19:30:26.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.633+0000 7f413ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:26.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.633+0000 7f413ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:26.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.937+0000 7f415d91c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:26.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.937+0000 7f415d91c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:26.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.937+0000 7f415d91c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:26.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.938+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41580b5ea0 con 0x7f41580b1730 2024-03-25T19:30:26.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.956+0000 7f413ffff640 1 -- 172.21.15.143:0/419780447 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f41480521e0 con 0x7f41580b1730 2024-03-25T19:30:26.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.956+0000 7f413ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:26.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.956+0000 7f413ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:26.957 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:30:26.957 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:26.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 10 librados: watch_flush enter 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 10 librados: watch_flush exit 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 10 client.4218.objecter successfully canceled tick 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 >> v1:172.21.15.143:6800/1292282666 conn(0x7f414406df20 legacy=0x7f41440703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 10 monclient: shutdown 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 >> v1:172.21.15.143:6789/0 conn(0x7f41580b1730 legacy=0x7f4158134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.959+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 shutdown_connections 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.960+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 >> 172.21.15.143:0/419780447 conn(0x7f41580617a0 msgr2=0x7f41580b6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.960+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 shutdown_connections 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.960+0000 7f415d91c640 1 -- 172.21.15.143:0/419780447 wait complete. 2024-03-25T19:30:26.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:26.960+0000 7f415d91c640 1 librados: shutdown 2024-03-25T19:30:26.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:28.059 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:00:26.235 35943== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:28.119 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:00:26.276 35946== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:28.228 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:00:26.405 35944== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:28.232 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:00:26.407 35945== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:28.840 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:00:26.993 35818== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:28.966 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:00:27.121 35817== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:29.521 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:00:27.699 35814== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:29.672 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:00:27.829 35816== Warning: unimplemented fcntl command: 1038 2024-03-25T19:30:33.779 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:33.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.878+0000 7fd90a36b640 10 monclient: get_monmap_and_config 2024-03-25T19:30:33.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.878+0000 7fd90a36b640 10 monclient: build_initial_monmap 2024-03-25T19:30:33.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.878+0000 7fd90a36b640 10 monclient: monmap: 2024-03-25T19:30:33.877 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:30:33.878998+0000 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:30:33.878998+0000 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:33.878 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.878+0000 7fd90a36b640 1 Processor -- start 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.878+0000 7fd90a36b640 1 -- start start 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.878+0000 7fd90a36b640 10 monclient: init 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 10 monclient(hunting): picked mon.noname-a con 0x7fd9040b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 10 monclient(hunting): picked mon.noname-c con 0x7fd9040b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 10 monclient(hunting): picked mon.noname-b con 0x7fd90414be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040af940 con 0x7fd9040b4d60 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040b0a70 con 0x7fd9040b1730 2024-03-25T19:30:33.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040b0d50 con 0x7fd90414be90 2024-03-25T19:30:33.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:33.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.879+0000 7fd90a36b640 10 monclient(hunting): authenticate will time out at 6050.223315s 2024-03-25T19:30:33.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.880+0000 7fd908908640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fd90414be90 0x7fd90414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:49626/0 (socket says 172.21.15.143:49626) 2024-03-25T19:30:33.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.880+0000 7fd908908640 1 -- 172.21.15.143:0/2337973358 learned_addr learned my addr 172.21.15.143:0/2337973358 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:33.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074246418 0 0) 0x7fd9040b0d50 con 0x7fd90414be90 2024-03-25T19:30:33.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ec003520 con 0x7fd90414be90 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713130946 0 0) 0x7fd9040b0a70 con 0x7fd9040b1730 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:30:33.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.883+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9040b0d50 con 0x7fd9040b1730 2024-03-25T19:30:33.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.884+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732687249 0 0) 0x7fd9040af940 con 0x7fd9040b4d60 2024-03-25T19:30:33.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.884+0000 7fd902ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.884+0000 7fd902ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.884+0000 7fd902ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.884+0000 7fd902ffd640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:30:33.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.884+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9040b0a70 con 0x7fd9040b4d60 2024-03-25T19:30:33.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955630437 0 0) 0x7fd8ec003520 con 0x7fd90414be90 2024-03-25T19:30:33.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 >> v1:172.21.15.203:6789/0 conn(0x7fd9040b1730 legacy=0x7fd9040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 >> v1:172.21.15.143:6789/0 conn(0x7fd9040b4d60 legacy=0x7fd90413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 1 monclient: found mon.noname-b 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9040b1030 con 0x7fd90414be90 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd8fc001a10 con 0x7fd90414be90 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd902ffd640 10 monclient: dump: 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:33.888 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd90a36b640 5 monclient: authenticate success, global_id 4214 2024-03-25T19:30:33.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.886+0000 7fd90a36b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:33.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.889+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8fc001b70 con 0x7fd90414be90 2024-03-25T19:30:33.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.889+0000 7fd902ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:33.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.889+0000 7fd902ffd640 1 -- 172.21.15.143:0/2337973358 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd8fc001f80 con 0x7fd90414be90 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.889+0000 7fd902ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.889+0000 7fd902ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.889+0000 7fd902ffd640 10 monclient: dump: 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.889+0000 7fd90a36b640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:33.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.889+0000 7fd90a36b640 4 set_mon_vals no callback set 2024-03-25T19:30:33.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.890+0000 7fd90a36b640 10 monclient: shutdown 2024-03-25T19:30:33.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.890+0000 7fd90a36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:33.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.890+0000 7fd90a36b640 1 -- 172.21.15.143:0/2337973358 >> v1:172.21.15.143:6790/0 conn(0x7fd90414be90 legacy=0x7fd90414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.891+0000 7fd90a36b640 1 -- 172.21.15.143:0/2337973358 shutdown_connections 2024-03-25T19:30:33.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.891+0000 7fd90a36b640 1 -- 172.21.15.143:0/2337973358 >> 172.21.15.143:0/2337973358 conn(0x7fd9040617a0 msgr2=0x7fd904061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:33.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.891+0000 7fd90a36b640 1 -- 172.21.15.143:0/2337973358 shutdown_connections 2024-03-25T19:30:33.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.891+0000 7fd90a36b640 1 -- 172.21.15.143:0/2337973358 wait complete. 2024-03-25T19:30:33.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.892+0000 7fd90a36b640 10 monclient: build_initial_monmap 2024-03-25T19:30:33.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.892+0000 7fd90a36b640 10 monclient: monmap: 2024-03-25T19:30:33.893 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:33.893 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.893+0000 7fd90a36b640 1 librados: starting msgr at 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.893+0000 7fd90a36b640 1 librados: starting objecter 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.893+0000 7fd90a36b640 1 Processor -- start 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.893+0000 7fd90a36b640 1 -- start start 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.893+0000 7fd90a36b640 1 librados: setting wanted keys 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.893+0000 7fd90a36b640 1 librados: calling monclient init 2024-03-25T19:30:33.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.893+0000 7fd90a36b640 10 monclient: init 2024-03-25T19:30:33.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.894+0000 7fd90a36b640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.894+0000 7fd90a36b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.894+0000 7fd90a36b640 10 monclient(hunting): picked mon.noname-c con 0x7fd9040b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.894+0000 7fd90a36b640 10 monclient(hunting): picked mon.noname-b con 0x7fd9040b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.894+0000 7fd90a36b640 10 monclient(hunting): picked mon.noname-a con 0x7fd90414be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd90a36b640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9041d79d0 con 0x7fd90414be90 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd90a36b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904137510 con 0x7fd9040b4d60 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd90a36b640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9041d8600 con 0x7fd9040b1730 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd90a36b640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd90a36b640 10 monclient(hunting): authenticate will time out at 6050.239031s 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd908908640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fd90414be90 0x7fd904138070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:33096/0 (socket says 172.21.15.143:33096) 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd908908640 1 -- 172.21.15.143:0/2407144349 learned_addr learned my addr 172.21.15.143:0/2407144349 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd900ff9640 10 client.?.objecter ms_handle_connect 0x7fd90414be90 2024-03-25T19:30:33.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd900ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:33.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd900ff9640 10 client.?.objecter ms_handle_connect 0x7fd9040b4d60 2024-03-25T19:30:33.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.895+0000 7fd900ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:33.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.896+0000 7fd900ff9640 10 client.?.objecter ms_handle_connect 0x7fd9040b1730 2024-03-25T19:30:33.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.896+0000 7fd900ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19754128 0 0) 0x7fd9041d79d0 con 0x7fd90414be90 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f0004eb0 con 0x7fd90414be90 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037717474 0 0) 0x7fd904137510 con 0x7fd9040b4d60 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 10 monclient(hunting): my global_id is 4228 2024-03-25T19:30:33.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.898+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9041d79d0 con 0x7fd9040b4d60 2024-03-25T19:30:33.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.899+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834368208 0 0) 0x7fd9041d8600 con 0x7fd9040b1730 2024-03-25T19:30:33.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.899+0000 7fd900ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:30:33.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.899+0000 7fd900ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:33.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.899+0000 7fd900ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:33.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.899+0000 7fd900ff9640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:30:33.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.899+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd904137510 con 0x7fd9040b1730 2024-03-25T19:30:33.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.903+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980077556 0 0) 0x7fd8f0004eb0 con 0x7fd90414be90 2024-03-25T19:30:33.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.903+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9041d8600 con 0x7fd90414be90 2024-03-25T19:30:33.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.903+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd8fc001ec0 con 0x7fd90414be90 2024-03-25T19:30:33.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.903+0000 7fd900ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.903+0000 7fd900ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.903+0000 7fd900ff9640 10 monclient(hunting): dump: 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:33.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.903+0000 7fd900ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:33.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45911220 0 0) 0x7fd9041d79d0 con 0x7fd9040b4d60 2024-03-25T19:30:33.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8f0004eb0 con 0x7fd9040b4d60 2024-03-25T19:30:33.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd8f8001970 con 0x7fd9040b4d60 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 10 monclient(hunting): dump: 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 813115982 0 0) 0x7fd9041d8600 con 0x7fd90414be90 2024-03-25T19:30:33.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 >> v1:172.21.15.143:6790/0 conn(0x7fd9040b1730 legacy=0x7fd904145300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 >> v1:172.21.15.203:6789/0 conn(0x7fd9040b4d60 legacy=0x7fd9041d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:33.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:30:33.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 1 monclient: found mon.a 2024-03-25T19:30:33.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:33.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9041d8790 con 0x7fd90414be90 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 10 monclient: _finish_auth 0 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd900ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 5 monclient: authenticate success, global_id 4224 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 10 monclient: _renew_subs 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd904138780 con 0x7fd90414be90 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 10 monclient: _renew_subs 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:33.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd904137b10 con 0x7fd90414be90 2024-03-25T19:30:33.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.906+0000 7fd90a36b640 1 librados: init done 2024-03-25T19:30:33.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.908+0000 7fd90a36b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:33.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.908+0000 7fd90a36b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:33.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.908+0000 7fd90a36b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:33.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:33.908+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd904138970 con 0x7fd90414be90 2024-03-25T19:30:34.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.132+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8fc004e10 con 0x7fd90414be90 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.132+0000 7fd900ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.132+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd8fc002100 con 0x7fd90414be90 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.132+0000 7fd900ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.132+0000 7fd900ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.132+0000 7fd900ff9640 10 monclient: dump: 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:34.132 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:34.133 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:34.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.132+0000 7fd9027fc640 4 set_mon_vals no callback set 2024-03-25T19:30:34.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.169+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fd8fc01af90 con 0x7fd90414be90 2024-03-25T19:30:34.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.170+0000 7fd903fff640 10 monclient: get_auth_request con 0x7fd8f006df70 auth_method 0 2024-03-25T19:30:34.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.170+0000 7fd900ff9640 10 client.4224.objecter ms_handle_connect 0x7fd8f006df70 2024-03-25T19:30:34.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.179+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fd8fc054c10 con 0x7fd90414be90 2024-03-25T19:30:34.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.179+0000 7fd900ff9640 10 client.4224.objecter ms_dispatch 0x7fd904000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:34.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.179+0000 7fd900ff9640 3 client.4224.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:34.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.179+0000 7fd900ff9640 3 client.4224.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:34.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.179+0000 7fd900ff9640 20 client.4224.objecter dump_active .. 0 homeless 2024-03-25T19:30:34.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.179+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8fc050550 con 0x7fd90414be90 2024-03-25T19:30:34.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.180+0000 7fd900ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:34.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.180+0000 7fd900ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.484+0000 7fd90a36b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.484+0000 7fd90a36b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.484+0000 7fd90a36b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:34.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.484+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9041d7b60 con 0x7fd90414be90 2024-03-25T19:30:34.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.508+0000 7fd900ff9640 1 -- 172.21.15.143:0/2407144349 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fd8fc052650 con 0x7fd90414be90 2024-03-25T19:30:34.508 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.508+0000 7fd900ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:34.508 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.508+0000 7fd900ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:34.508 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:30:34.508 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:34.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 10 librados: watch_flush enter 2024-03-25T19:30:34.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 10 librados: watch_flush exit 2024-03-25T19:30:34.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 20 client.4224.objecter shutdown clearing up homeless session... 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 10 client.4224.objecter successfully canceled tick 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 >> v1:172.21.15.143:6800/1292282666 conn(0x7fd8f006df70 legacy=0x7fd8f0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 10 monclient: shutdown 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 >> v1:172.21.15.143:6789/0 conn(0x7fd90414be90 legacy=0x7fd904138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 shutdown_connections 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.510+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 >> 172.21.15.143:0/2407144349 conn(0x7fd9040617a0 msgr2=0x7fd9040b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.511+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 shutdown_connections 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.511+0000 7fd90a36b640 1 -- 172.21.15.143:0/2407144349 wait complete. 2024-03-25T19:30:34.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:34.511+0000 7fd90a36b640 1 librados: shutdown 2024-03-25T19:30:34.524 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:36.747 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-25T19:30:36.745+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:30:36.797 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-25T19:30:36.794+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:30:36.831 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-25T19:30:36.797+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:30:37.147 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-25T19:30:37.097+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:30:37.644 INFO:tasks.ceph.osd.2.smithi143.stderr:2024-03-25T19:30:37.642+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:30:37.811 INFO:tasks.ceph.osd.1.smithi143.stderr:2024-03-25T19:30:37.809+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:30:38.931 INFO:tasks.ceph.osd.0.smithi143.stderr:2024-03-25T19:30:38.929+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:30:38.958 INFO:tasks.ceph.osd.3.smithi143.stderr:2024-03-25T19:30:38.955+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:30:41.330 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:41.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.448+0000 7f49fa001640 10 monclient: get_monmap_and_config 2024-03-25T19:30:41.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.449+0000 7f49fa001640 10 monclient: build_initial_monmap 2024-03-25T19:30:41.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.449+0000 7f49fa001640 10 monclient: monmap: 2024-03-25T19:30:41.448 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:30:41.449931+0000 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:30:41.449931+0000 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:41.449 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:41.450 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.449+0000 7f49fa001640 1 Processor -- start 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.449+0000 7f49fa001640 1 -- start start 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.449+0000 7f49fa001640 10 monclient: init 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.450+0000 7f49fa001640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.450+0000 7f49fa001640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.450+0000 7f49fa001640 10 monclient(hunting): picked mon.noname-a con 0x7f49f40a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.450+0000 7f49fa001640 10 monclient(hunting): picked mon.noname-b con 0x7f49f413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.450+0000 7f49fa001640 10 monclient(hunting): picked mon.noname-c con 0x7f49f41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.450+0000 7f49fa001640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f4142520 con 0x7f49f40a98f0 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.451+0000 7f49fa001640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f40b28a0 con 0x7f49f41430c0 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.451+0000 7f49fa001640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f40b2a30 con 0x7f49f413f710 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.451+0000 7f49fa001640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:41.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.451+0000 7f49fa001640 10 monclient(hunting): authenticate will time out at 6057.794682s 2024-03-25T19:30:41.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.453+0000 7f49f3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f49f41430c0 0x7f49f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:59784/0 (socket says 172.21.15.143:59784) 2024-03-25T19:30:41.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.453+0000 7f49f3fff640 1 -- 172.21.15.143:0/1767208117 learned_addr learned my addr 172.21.15.143:0/1767208117 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:41.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.472+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058826549 0 0) 0x7f49f40b28a0 con 0x7f49f41430c0 2024-03-25T19:30:41.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.472+0000 7f49f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.472+0000 7f49f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.472+0000 7f49f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.472+0000 7f49f27fc640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:30:41.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.472+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49dc003520 con 0x7f49f41430c0 2024-03-25T19:30:41.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.475+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407229296 0 0) 0x7f49f40b2a30 con 0x7f49f413f710 2024-03-25T19:30:41.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.475+0000 7f49f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.475+0000 7f49f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.475+0000 7f49f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.475+0000 7f49f27fc640 10 monclient(hunting): my global_id is 4244 2024-03-25T19:30:41.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.475+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f40b28a0 con 0x7f49f413f710 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3150031763 0 0) 0x7f49f40b28a0 con 0x7f49f413f710 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 >> v1:172.21.15.203:6789/0 conn(0x7f49f41430c0 legacy=0x7f49f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 >> v1:172.21.15.143:6789/0 conn(0x7f49f40a98f0 legacy=0x7f49f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 1 monclient: found mon.noname-b 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f40b44c0 con 0x7f49f413f710 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:41.484 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f49e8001670 con 0x7f49f413f710 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49f27fc640 10 monclient: dump: 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49fa001640 5 monclient: authenticate success, global_id 4244 2024-03-25T19:30:41.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.484+0000 7f49fa001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:41.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.490+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49e80017d0 con 0x7f49f413f710 2024-03-25T19:30:41.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.490+0000 7f49f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.490+0000 7f49f27fc640 1 -- 172.21.15.143:0/1767208117 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f49e8001c20 con 0x7f49f413f710 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.490+0000 7f49f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.490+0000 7f49f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.490+0000 7f49f27fc640 10 monclient: dump: 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:41.491 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:41.492 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:41.492 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:41.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.490+0000 7f49fa001640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:41.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.490+0000 7f49fa001640 4 set_mon_vals no callback set 2024-03-25T19:30:41.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.491+0000 7f49fa001640 10 monclient: shutdown 2024-03-25T19:30:41.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.491+0000 7f49fa001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:41.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.491+0000 7f49fa001640 1 -- 172.21.15.143:0/1767208117 >> v1:172.21.15.143:6790/0 conn(0x7f49f413f710 legacy=0x7f49f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.491+0000 7f49fa001640 1 -- 172.21.15.143:0/1767208117 shutdown_connections 2024-03-25T19:30:41.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.491+0000 7f49fa001640 1 -- 172.21.15.143:0/1767208117 >> 172.21.15.143:0/1767208117 conn(0x7f49f40b7ea0 msgr2=0x7f49f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:41.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.491+0000 7f49fa001640 1 -- 172.21.15.143:0/1767208117 shutdown_connections 2024-03-25T19:30:41.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.491+0000 7f49fa001640 1 -- 172.21.15.143:0/1767208117 wait complete. 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.492+0000 7f49fa001640 10 monclient: build_initial_monmap 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.492+0000 7f49fa001640 10 monclient: monmap: 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.493+0000 7f49fa001640 1 librados: starting msgr at 2024-03-25T19:30:41.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.493+0000 7f49fa001640 1 librados: starting objecter 2024-03-25T19:30:41.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.493+0000 7f49fa001640 1 Processor -- start 2024-03-25T19:30:41.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.493+0000 7f49fa001640 1 -- start start 2024-03-25T19:30:41.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.493+0000 7f49fa001640 1 librados: setting wanted keys 2024-03-25T19:30:41.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.493+0000 7f49fa001640 1 librados: calling monclient init 2024-03-25T19:30:41.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.493+0000 7f49fa001640 10 monclient: init 2024-03-25T19:30:41.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:41.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:41.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 10 monclient(hunting): picked mon.noname-c con 0x7f49f40a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:41.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 10 monclient(hunting): picked mon.noname-a con 0x7f49f413f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:41.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 10 monclient(hunting): picked mon.noname-b con 0x7f49f41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:41.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f41d78e0 con 0x7f49f413f710 2024-03-25T19:30:41.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f41d7500 con 0x7f49f41430c0 2024-03-25T19:30:41.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f41d7690 con 0x7f49f40a98f0 2024-03-25T19:30:41.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:41.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.494+0000 7f49fa001640 10 monclient(hunting): authenticate will time out at 6057.838302s 2024-03-25T19:30:41.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.503+0000 7f49f3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f49f41430c0 0x7f49f4138560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:59818/0 (socket says 172.21.15.143:59818) 2024-03-25T19:30:41.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.503+0000 7f49f3fff640 1 -- 172.21.15.143:0/113791869 learned_addr learned my addr 172.21.15.143:0/113791869 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:41.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.504+0000 7f49dbfff640 10 client.?.objecter ms_handle_connect 0x7f49f41430c0 2024-03-25T19:30:41.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.504+0000 7f49dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:41.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.514+0000 7f49dbfff640 10 client.?.objecter ms_handle_connect 0x7f49f40a98f0 2024-03-25T19:30:41.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.514+0000 7f49dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:41.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.514+0000 7f49dbfff640 10 client.?.objecter ms_handle_connect 0x7f49f413f710 2024-03-25T19:30:41.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.514+0000 7f49dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:41.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.518+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056217683 0 0) 0x7f49f41d7500 con 0x7f49f41430c0 2024-03-25T19:30:41.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.518+0000 7f49dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.518+0000 7f49dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.518+0000 7f49dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.518+0000 7f49dbfff640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:30:41.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.518+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e0004eb0 con 0x7f49f41430c0 2024-03-25T19:30:41.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.520+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641499436 0 0) 0x7f49f41d7690 con 0x7f49f40a98f0 2024-03-25T19:30:41.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.520+0000 7f49dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.520+0000 7f49dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.520+0000 7f49dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.520+0000 7f49dbfff640 10 monclient(hunting): my global_id is 4247 2024-03-25T19:30:41.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.520+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f41d7500 con 0x7f49f40a98f0 2024-03-25T19:30:41.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.534+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2253049525 0 0) 0x7f49e0004eb0 con 0x7f49f41430c0 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.534+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49f41d7690 con 0x7f49f41430c0 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.534+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f49ec001c10 con 0x7f49f41430c0 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.534+0000 7f49dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.534+0000 7f49dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.534+0000 7f49dbfff640 10 monclient(hunting): dump: 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.534 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:41.535 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:41.535 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:41.535 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:41.535 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:41.535 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.534+0000 7f49dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:41.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.538+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742328183 0 0) 0x7f49f41d78e0 con 0x7f49f413f710 2024-03-25T19:30:41.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.538+0000 7f49dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:41.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.538+0000 7f49dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:41.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.538+0000 7f49dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:41.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.538+0000 7f49dbfff640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:30:41.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.538+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e0004eb0 con 0x7f49f413f710 2024-03-25T19:30:41.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.617+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4160584537 0 0) 0x7f49e0004eb0 con 0x7f49f413f710 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.617+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49f41d78e0 con 0x7f49f413f710 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.617+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f49e8001ae0 con 0x7f49f413f710 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.617+0000 7f49dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.617+0000 7f49dbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.617+0000 7f49dbfff640 10 monclient(hunting): dump: 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:41.617 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:41.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 447783070 0 0) 0x7f49f41d78e0 con 0x7f49f413f710 2024-03-25T19:30:41.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 >> v1:172.21.15.143:6790/0 conn(0x7f49f40a98f0 legacy=0x7f49f4134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 >> v1:172.21.15.203:6789/0 conn(0x7f49f41430c0 legacy=0x7f49f4138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:41.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:41.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 1 monclient: found mon.a 2024-03-25T19:30:41.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:41.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f41377c0 con 0x7f49f413f710 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 10 monclient: _finish_auth 0 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 5 monclient: authenticate success, global_id 4254 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 10 monclient: _renew_subs 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f4137a70 con 0x7f49f413f710 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 10 monclient: _renew_subs 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f41341f0 con 0x7f49f413f710 2024-03-25T19:30:41.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.633+0000 7f49fa001640 1 librados: init done 2024-03-25T19:30:41.635 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.635+0000 7f49fa001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:41.635 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.635+0000 7f49fa001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:41.635 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.635+0000 7f49fa001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:41.635 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.635+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f40abbe0 con 0x7f49f413f710 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.924+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49e80038a0 con 0x7f49f413f710 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.924+0000 7f49dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.924+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f49e8001ce0 con 0x7f49f413f710 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.924+0000 7f49dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.924+0000 7f49dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.924+0000 7f49dbfff640 10 monclient: dump: 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:41.924 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:41.925 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:41.925 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:41.925 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:41.925 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:41.925 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:41.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.924+0000 7f49f1ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:41.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.925+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f49e8019a20 con 0x7f49f413f710 2024-03-25T19:30:41.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.925+0000 7f49f37fe640 10 monclient: get_auth_request con 0x7f49e006df20 auth_method 0 2024-03-25T19:30:41.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.926+0000 7f49dbfff640 10 client.4254.objecter ms_handle_connect 0x7f49e006df20 2024-03-25T19:30:41.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.935+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f49e8053620 con 0x7f49f413f710 2024-03-25T19:30:41.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.935+0000 7f49dbfff640 10 client.4254.objecter ms_dispatch 0x7f49f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:41.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.935+0000 7f49dbfff640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:41.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.935+0000 7f49dbfff640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:41.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.935+0000 7f49dbfff640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-25T19:30:41.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.935+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49e804ef60 con 0x7f49f413f710 2024-03-25T19:30:41.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.935+0000 7f49dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:41.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:41.935+0000 7f49dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:42.187 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.188+0000 7f49fa001640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.188 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.188+0000 7f49fa001640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.188 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.188+0000 7f49fa001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:42.188 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.188+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49f40627b0 con 0x7f49f413f710 2024-03-25T19:30:42.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.235+0000 7f49dbfff640 1 -- 172.21.15.143:0/113791869 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f49e8051060 con 0x7f49f413f710 2024-03-25T19:30:42.235 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.235+0000 7f49dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:42.235 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.235+0000 7f49dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:42.235 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:30:42.235 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:42.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.237+0000 7f49fa001640 10 librados: watch_flush enter 2024-03-25T19:30:42.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.237+0000 7f49fa001640 10 librados: watch_flush exit 2024-03-25T19:30:42.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.237+0000 7f49fa001640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.237+0000 7f49fa001640 10 client.4254.objecter successfully canceled tick 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.237+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 >> v1:172.21.15.143:6800/1292282666 conn(0x7f49e006df20 legacy=0x7f49e00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.237+0000 7f49fa001640 10 monclient: shutdown 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.237+0000 7f49fa001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.237+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 >> v1:172.21.15.143:6789/0 conn(0x7f49f413f710 legacy=0x7f49f41d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.238+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 shutdown_connections 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.238+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 >> 172.21.15.143:0/113791869 conn(0x7f49f40b7ea0 msgr2=0x7f49f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.238+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 shutdown_connections 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.238+0000 7f49fa001640 1 -- 172.21.15.143:0/113791869 wait complete. 2024-03-25T19:30:42.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:42.238+0000 7f49fa001640 1 librados: shutdown 2024-03-25T19:30:42.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:49.057 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:49.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.177+0000 7f1e60932640 10 monclient: get_monmap_and_config 2024-03-25T19:30:49.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.177+0000 7f1e60932640 10 monclient: build_initial_monmap 2024-03-25T19:30:49.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.177+0000 7f1e60932640 10 monclient: monmap: 2024-03-25T19:30:49.177 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:49.177 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:49.177 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:30:49.178486+0000 2024-03-25T19:30:49.177 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:30:49.178486+0000 2024-03-25T19:30:49.178 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:49.178 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:49.178 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:49.178 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:49.178 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:49.178 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:49.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.178+0000 7f1e60932640 1 Processor -- start 2024-03-25T19:30:49.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.178+0000 7f1e60932640 1 -- start start 2024-03-25T19:30:49.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.178+0000 7f1e60932640 10 monclient: init 2024-03-25T19:30:49.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.178+0000 7f1e60932640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.178+0000 7f1e60932640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.179+0000 7f1e60932640 10 monclient(hunting): picked mon.noname-b con 0x7f1e5c0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.179+0000 7f1e60932640 10 monclient(hunting): picked mon.noname-a con 0x7f1e5c13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.179+0000 7f1e60932640 10 monclient(hunting): picked mon.noname-c con 0x7f1e5c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.179+0000 7f1e60932640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c142520 con 0x7f1e5c13f710 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.179+0000 7f1e60932640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c0b28a0 con 0x7f1e5c1430c0 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.179+0000 7f1e60932640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c0b2a30 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.179+0000 7f1e60932640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:49.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.179+0000 7f1e60932640 10 monclient(hunting): authenticate will time out at 6065.522740s 2024-03-25T19:30:49.188 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.189+0000 7f1e5a59c640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f1e5c0a98f0 0x7f1e5c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:57550/0 (socket says 172.21.15.143:57550) 2024-03-25T19:30:49.188 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.189+0000 7f1e5a59c640 1 -- 172.21.15.143:0/2754770204 learned_addr learned my addr 172.21.15.143:0/2754770204 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:49.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.204+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200798801 0 0) 0x7f1e5c0b2a30 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.204+0000 7f1e59d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.204+0000 7f1e59d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.204+0000 7f1e59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.205 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.204+0000 7f1e59d9b640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:30:49.205 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.205+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e48003520 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.206+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812584045 0 0) 0x7f1e5c0b28a0 con 0x7f1e5c1430c0 2024-03-25T19:30:49.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.206+0000 7f1e59d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.206+0000 7f1e59d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.206+0000 7f1e59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.206+0000 7f1e59d9b640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:30:49.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.206+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e5c0b2a30 con 0x7f1e5c1430c0 2024-03-25T19:30:49.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.210+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843882010 0 0) 0x7f1e5c142520 con 0x7f1e5c13f710 2024-03-25T19:30:49.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.210+0000 7f1e59d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.210+0000 7f1e59d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.210+0000 7f1e59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.210+0000 7f1e59d9b640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:30:49.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.210+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e5c0b28a0 con 0x7f1e5c13f710 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1916938312 0 0) 0x7f1e48003520 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 >> v1:172.21.15.203:6789/0 conn(0x7f1e5c1430c0 legacy=0x7f1e5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 >> v1:172.21.15.143:6789/0 conn(0x7f1e5c13f710 legacy=0x7f1e5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 1 monclient: found mon.noname-b 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c0b44c0 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 10 monclient: _finish_auth 0 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1e4c001690 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e59d9b640 10 monclient: dump: 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:49.222 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:49.223 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:49.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e60932640 5 monclient: authenticate success, global_id 4250 2024-03-25T19:30:49.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.221+0000 7f1e60932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:49.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.243+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e4c0017f0 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.243+0000 7f1e59d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:49.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.243+0000 7f1e59d9b640 1 -- 172.21.15.143:0/2754770204 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1e4c001c40 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.243+0000 7f1e59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.243+0000 7f1e59d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.243+0000 7f1e59d9b640 10 monclient: dump: 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:49.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.243+0000 7f1e60932640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:49.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.243+0000 7f1e60932640 4 set_mon_vals no callback set 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.244+0000 7f1e60932640 10 monclient: shutdown 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.244+0000 7f1e60932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.244+0000 7f1e60932640 1 -- 172.21.15.143:0/2754770204 >> v1:172.21.15.143:6790/0 conn(0x7f1e5c0a98f0 legacy=0x7f1e5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.244+0000 7f1e60932640 1 -- 172.21.15.143:0/2754770204 shutdown_connections 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.244+0000 7f1e60932640 1 -- 172.21.15.143:0/2754770204 >> 172.21.15.143:0/2754770204 conn(0x7f1e5c0b7ea0 msgr2=0x7f1e5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.245+0000 7f1e60932640 1 -- 172.21.15.143:0/2754770204 shutdown_connections 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.245+0000 7f1e60932640 1 -- 172.21.15.143:0/2754770204 wait complete. 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 10 monclient: build_initial_monmap 2024-03-25T19:30:49.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 10 monclient: monmap: 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 1 librados: starting msgr at 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 1 librados: starting objecter 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 1 Processor -- start 2024-03-25T19:30:49.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 1 -- start start 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 1 librados: setting wanted keys 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 1 librados: calling monclient init 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 10 monclient: init 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.246+0000 7f1e60932640 10 monclient(hunting): picked mon.noname-c con 0x7f1e5c0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.247+0000 7f1e60932640 10 monclient(hunting): picked mon.noname-a con 0x7f1e5c13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.247+0000 7f1e60932640 10 monclient(hunting): picked mon.noname-b con 0x7f1e5c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.247+0000 7f1e60932640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c0b1850 con 0x7f1e5c13f710 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.247+0000 7f1e60932640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c0abaa0 con 0x7f1e5c1430c0 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.247+0000 7f1e60932640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5c0abd10 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.247+0000 7f1e60932640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:49.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.247+0000 7f1e60932640 10 monclient(hunting): authenticate will time out at 6065.590706s 2024-03-25T19:30:49.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.259+0000 7f1e53fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1e5c13f710 0x7f1e5c0ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:42464/0 (socket says 172.21.15.143:42464) 2024-03-25T19:30:49.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.259+0000 7f1e53fff640 1 -- 172.21.15.143:0/2791474157 learned_addr learned my addr 172.21.15.143:0/2791474157 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:49.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.260+0000 7f1e52ffd640 10 client.?.objecter ms_handle_connect 0x7f1e5c13f710 2024-03-25T19:30:49.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.260+0000 7f1e52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.263 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.263+0000 7f1e52ffd640 10 client.?.objecter ms_handle_connect 0x7f1e5c1430c0 2024-03-25T19:30:49.263 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.263+0000 7f1e52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.266+0000 7f1e52ffd640 10 client.?.objecter ms_handle_connect 0x7f1e5c0a98f0 2024-03-25T19:30:49.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.266+0000 7f1e52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:49.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.266+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584369997 0 0) 0x7f1e5c0b1850 con 0x7f1e5c13f710 2024-03-25T19:30:49.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.266+0000 7f1e52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.266+0000 7f1e52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.266+0000 7f1e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.266+0000 7f1e52ffd640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:30:49.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.266+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e3c004eb0 con 0x7f1e5c13f710 2024-03-25T19:30:49.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.277+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159872965 0 0) 0x7f1e5c0abd10 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.277+0000 7f1e52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.277+0000 7f1e52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.277+0000 7f1e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.277+0000 7f1e52ffd640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:30:49.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.277+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e5c0b1850 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.278+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469116947 0 0) 0x7f1e5c0abaa0 con 0x7f1e5c1430c0 2024-03-25T19:30:49.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.278+0000 7f1e52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:49.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.278+0000 7f1e52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:49.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.278+0000 7f1e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:49.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.278+0000 7f1e52ffd640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:30:49.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.278+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e5c0abd10 con 0x7f1e5c1430c0 2024-03-25T19:30:49.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.284+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2571010066 0 0) 0x7f1e3c004eb0 con 0x7f1e5c13f710 2024-03-25T19:30:49.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.284+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e5c0abaa0 con 0x7f1e5c13f710 2024-03-25T19:30:49.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.284+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1e44001a10 con 0x7f1e5c13f710 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.284+0000 7f1e52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.284+0000 7f1e52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.284+0000 7f1e52ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:49.285 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:49.286 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:49.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.285+0000 7f1e52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:49.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.288+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1748270564 0 0) 0x7f1e5c0b1850 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.288+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e3c004eb0 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.288+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1e4c001b00 con 0x7f1e5c0a98f0 2024-03-25T19:30:49.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.288+0000 7f1e52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.288+0000 7f1e52ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:49.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.288+0000 7f1e52ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:49.288 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:49.289 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:49.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.293+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335594538 0 0) 0x7f1e5c0abd10 con 0x7f1e5c1430c0 2024-03-25T19:30:49.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.293+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e5c0b1850 con 0x7f1e5c1430c0 2024-03-25T19:30:49.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.298+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1e54001bd0 con 0x7f1e5c1430c0 2024-03-25T19:30:49.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.298+0000 7f1e52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.298+0000 7f1e52ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:49.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.298+0000 7f1e52ffd640 10 monclient(hunting): dump: 2024-03-25T19:30:49.298 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:49.298 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:49.298 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.298 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.299 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.299 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:49.299 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:49.299 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:49.299 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:49.299 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:49.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3680887117 0 0) 0x7f1e5c0abaa0 con 0x7f1e5c13f710 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 >> v1:172.21.15.143:6790/0 conn(0x7f1e5c0a98f0 legacy=0x7f1e5c1387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 >> v1:172.21.15.203:6789/0 conn(0x7f1e5c1430c0 legacy=0x7f1e5c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 1 monclient: found mon.a 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c0abff0 con 0x7f1e5c13f710 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 10 monclient: _finish_auth 0 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 5 monclient: authenticate success, global_id 4260 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 10 monclient: _renew_subs 2024-03-25T19:30:49.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5c0acb20 con 0x7f1e5c13f710 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 10 monclient: _renew_subs 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5c1364e0 con 0x7f1e5c13f710 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.299+0000 7f1e60932640 1 librados: init done 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.300+0000 7f1e60932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.300+0000 7f1e60932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.300+0000 7f1e60932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:49.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.300+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c0627b0 con 0x7f1e5c13f710 2024-03-25T19:30:49.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.524+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e44001c70 con 0x7f1e5c13f710 2024-03-25T19:30:49.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.524+0000 7f1e52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:49.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.524+0000 7f1e5959a640 4 set_mon_vals no callback set 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.615+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1e44003a60 con 0x7f1e5c13f710 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.615+0000 7f1e52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.615+0000 7f1e52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.615+0000 7f1e52ffd640 10 monclient: dump: 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:49.615 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:49.616 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:49.616 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:49.616 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:49.616 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:49.616 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:49.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.616+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f1e44003bc0 con 0x7f1e5c13f710 2024-03-25T19:30:49.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.617+0000 7f1e5a59c640 10 monclient: get_auth_request con 0x7f1e3c06e090 auth_method 0 2024-03-25T19:30:49.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.617+0000 7f1e52ffd640 10 client.4260.objecter ms_handle_connect 0x7f1e3c06e090 2024-03-25T19:30:49.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.623+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f1e440548c0 con 0x7f1e5c13f710 2024-03-25T19:30:49.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.623+0000 7f1e52ffd640 10 client.4260.objecter ms_dispatch 0x7f1e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:49.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.623+0000 7f1e52ffd640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:49.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.623+0000 7f1e52ffd640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:49.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.623+0000 7f1e52ffd640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-25T19:30:49.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.623+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e440503d0 con 0x7f1e5c13f710 2024-03-25T19:30:49.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.623+0000 7f1e52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:49.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.623+0000 7f1e52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:49.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.869+0000 7f1e60932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:49.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.869+0000 7f1e60932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:49.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.869+0000 7f1e60932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:30:49.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.869+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e5c13dbb0 con 0x7f1e5c13f710 2024-03-25T19:30:49.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.896+0000 7f1e52ffd640 1 -- 172.21.15.143:0/2791474157 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f1e44052300 con 0x7f1e5c13f710 2024-03-25T19:30:49.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.896+0000 7f1e52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:49.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.896+0000 7f1e52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:49.896 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:30:49.896 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:49.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 10 librados: watch_flush enter 2024-03-25T19:30:49.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 10 librados: watch_flush exit 2024-03-25T19:30:49.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-25T19:30:49.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 10 client.4260.objecter successfully canceled tick 2024-03-25T19:30:49.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 >> v1:172.21.15.143:6800/1292282666 conn(0x7f1e3c06e090 legacy=0x7f1e3c070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 10 monclient: shutdown 2024-03-25T19:30:49.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:49.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 >> v1:172.21.15.143:6789/0 conn(0x7f1e5c13f710 legacy=0x7f1e5c0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:49.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 shutdown_connections 2024-03-25T19:30:49.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 >> 172.21.15.143:0/2791474157 conn(0x7f1e5c0b7ea0 msgr2=0x7f1e5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:49.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 shutdown_connections 2024-03-25T19:30:49.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 1 -- 172.21.15.143:0/2791474157 wait complete. 2024-03-25T19:30:49.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:49.899+0000 7f1e60932640 1 librados: shutdown 2024-03-25T19:30:49.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:52.700 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:19.265 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.764 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:19.328 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.797 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:19.362 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.868 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:19.433 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.903 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:19.438 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.968 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:19.504 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:52.996 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:19.561 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:53.102 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:19.667 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:53.184 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:19.720 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:56.716 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:56.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.829+0000 7f08dd006640 10 monclient: get_monmap_and_config 2024-03-25T19:30:56.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.829+0000 7f08dd006640 10 monclient: build_initial_monmap 2024-03-25T19:30:56.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.829+0000 7f08dd006640 10 monclient: monmap: 2024-03-25T19:30:56.829 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:56.829 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:56.829 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:30:56.830395+0000 2024-03-25T19:30:56.829 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:30:56.830395+0000 2024-03-25T19:30:56.830 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:56.830 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:56.830 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:56.830 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:30:56.830 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:30:56.830 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:56.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.830+0000 7f08dd006640 1 Processor -- start 2024-03-25T19:30:56.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.830+0000 7f08dd006640 1 -- start start 2024-03-25T19:30:56.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.830+0000 7f08dd006640 10 monclient: init 2024-03-25T19:30:56.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.830+0000 7f08dd006640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:56.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.830+0000 7f08dd006640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:56.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.830+0000 7f08dd006640 10 monclient(hunting): picked mon.noname-b con 0x7f08d80a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:56.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.831+0000 7f08dd006640 10 monclient(hunting): picked mon.noname-a con 0x7f08d813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:56.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.831+0000 7f08dd006640 10 monclient(hunting): picked mon.noname-c con 0x7f08d81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:56.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.831+0000 7f08dd006640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d8142520 con 0x7f08d813f710 2024-03-25T19:30:56.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.831+0000 7f08dd006640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80b28a0 con 0x7f08d81430c0 2024-03-25T19:30:56.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.831+0000 7f08dd006640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80b2a30 con 0x7f08d80a98f0 2024-03-25T19:30:56.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.831+0000 7f08dd006640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:56.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.831+0000 7f08dd006640 10 monclient(hunting): authenticate will time out at 6073.174850s 2024-03-25T19:30:56.841 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.842+0000 7f08d659c640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f08d813f710 0x7f08d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:56852/0 (socket says 172.21.15.143:56852) 2024-03-25T19:30:56.841 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.842+0000 7f08d659c640 1 -- 172.21.15.143:0/1765517895 learned_addr learned my addr 172.21.15.143:0/1765517895 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:56.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.853+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216451300 0 0) 0x7f08d8142520 con 0x7f08d813f710 2024-03-25T19:30:56.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.853+0000 7f08d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.853+0000 7f08d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.853+0000 7f08d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.853+0000 7f08d5d9b640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:30:56.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.853+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c0003520 con 0x7f08d813f710 2024-03-25T19:30:56.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.859+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292633468 0 0) 0x7f08d80b2a30 con 0x7f08d80a98f0 2024-03-25T19:30:56.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.859+0000 7f08d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.859+0000 7f08d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.859+0000 7f08d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.859+0000 7f08d5d9b640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:30:56.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.859+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d8142520 con 0x7f08d80a98f0 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 763380608 0 0) 0x7f08c0003520 con 0x7f08d813f710 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 >> v1:172.21.15.143:6790/0 conn(0x7f08d80a98f0 legacy=0x7f08d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 >> v1:172.21.15.203:6789/0 conn(0x7f08d81430c0 legacy=0x7f08d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 1 monclient: found mon.noname-a 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d80b44c0 con 0x7f08d813f710 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:30:56.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f08cc001690 con 0x7f08d813f710 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08d5d9b640 10 monclient: dump: 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:56.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08dd006640 5 monclient: authenticate success, global_id 4263 2024-03-25T19:30:56.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.860+0000 7f08dd006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.866+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08cc0017f0 con 0x7f08d813f710 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.866+0000 7f08d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.866+0000 7f08d5d9b640 1 -- 172.21.15.143:0/1765517895 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f08cc001c00 con 0x7f08d813f710 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.866+0000 7f08dd006640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.866+0000 7f08dd006640 4 set_mon_vals no callback set 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.866+0000 7f08d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.866+0000 7f08d5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.866+0000 7f08d5d9b640 10 monclient: dump: 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:56.866 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.867 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:56.867 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:56.867 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:56.867 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:56.867 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:56.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.867+0000 7f08dd006640 10 monclient: shutdown 2024-03-25T19:30:56.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.867+0000 7f08dd006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:56.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.867+0000 7f08dd006640 1 -- 172.21.15.143:0/1765517895 >> v1:172.21.15.143:6789/0 conn(0x7f08d813f710 legacy=0x7f08d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.867+0000 7f08dd006640 1 -- 172.21.15.143:0/1765517895 shutdown_connections 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.867+0000 7f08dd006640 1 -- 172.21.15.143:0/1765517895 >> 172.21.15.143:0/1765517895 conn(0x7f08d80b7ea0 msgr2=0x7f08d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.867+0000 7f08dd006640 1 -- 172.21.15.143:0/1765517895 shutdown_connections 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.867+0000 7f08dd006640 1 -- 172.21.15.143:0/1765517895 wait complete. 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.868+0000 7f08dd006640 10 monclient: build_initial_monmap 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.868+0000 7f08dd006640 10 monclient: monmap: 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:30:56.869 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:30:56.870 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:56.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.868+0000 7f08dd006640 1 librados: starting msgr at 2024-03-25T19:30:56.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.868+0000 7f08dd006640 1 librados: starting objecter 2024-03-25T19:30:56.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 1 Processor -- start 2024-03-25T19:30:56.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 1 -- start start 2024-03-25T19:30:56.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 1 librados: setting wanted keys 2024-03-25T19:30:56.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 1 librados: calling monclient init 2024-03-25T19:30:56.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 10 monclient: init 2024-03-25T19:30:56.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:56.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:56.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 10 monclient(hunting): picked mon.noname-a con 0x7f08d80a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:30:56.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 10 monclient(hunting): picked mon.noname-c con 0x7f08d813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.869+0000 7f08dd006640 10 monclient(hunting): picked mon.noname-b con 0x7f08d81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.870+0000 7f08dd006640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d8137f10 con 0x7f08d80a98f0 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.870+0000 7f08dd006640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d81dc070 con 0x7f08d81430c0 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.870+0000 7f08dd006640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d8137100 con 0x7f08d813f710 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.870+0000 7f08dd006640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.870+0000 7f08dd006640 10 monclient(hunting): authenticate will time out at 6073.213843s 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.871+0000 7f08d6d9d640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f08d80a98f0 0x7f08d8137800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:56880/0 (socket says 172.21.15.143:56880) 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.871+0000 7f08d6d9d640 1 -- 172.21.15.143:0/3284339354 learned_addr learned my addr 172.21.15.143:0/3284339354 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.872+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d80a98f0 2024-03-25T19:30:56.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.872+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:56.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.888+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d81430c0 2024-03-25T19:30:56.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.888+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:56.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.894+0000 7f08bf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d813f710 2024-03-25T19:30:56.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.894+0000 7f08bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:56.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.895+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232483269 0 0) 0x7f08d8137f10 con 0x7f08d80a98f0 2024-03-25T19:30:56.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.895+0000 7f08bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.895+0000 7f08bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.895+0000 7f08bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.895+0000 7f08bf7fe640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:30:56.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.895+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c4004eb0 con 0x7f08d80a98f0 2024-03-25T19:30:56.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.896+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006603159 0 0) 0x7f08d81dc070 con 0x7f08d81430c0 2024-03-25T19:30:56.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.896+0000 7f08bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.896+0000 7f08bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.896+0000 7f08bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.896+0000 7f08bf7fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:30:56.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.896+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d8137f10 con 0x7f08d81430c0 2024-03-25T19:30:56.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.916+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912524759 0 0) 0x7f08d8137100 con 0x7f08d813f710 2024-03-25T19:30:56.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.916+0000 7f08bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:56.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.916+0000 7f08bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:56.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.916+0000 7f08bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:56.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.916+0000 7f08bf7fe640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:30:56.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.916+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d81dc070 con 0x7f08d813f710 2024-03-25T19:30:56.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.919+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2355240228 0 0) 0x7f08c4004eb0 con 0x7f08d80a98f0 2024-03-25T19:30:56.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.919+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08d8137100 con 0x7f08d80a98f0 2024-03-25T19:30:56.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.919+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f08c8001a10 con 0x7f08d80a98f0 2024-03-25T19:30:56.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.919+0000 7f08bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.919+0000 7f08bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:56.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.919+0000 7f08bf7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:56.919 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:56.919 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.919+0000 7f08bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.919+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 908032410 0 0) 0x7f08d8137f10 con 0x7f08d81430c0 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.920+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08c4004eb0 con 0x7f08d81430c0 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.920+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f08d0001b00 con 0x7f08d81430c0 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.920+0000 7f08bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.920+0000 7f08bf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:56.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.920+0000 7f08bf7fe640 10 monclient(hunting): dump: 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:56.921 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:56.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3739123259 0 0) 0x7f08c4004eb0 con 0x7f08d81430c0 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 >> v1:172.21.15.143:6790/0 conn(0x7f08d813f710 legacy=0x7f08d81db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 >> v1:172.21.15.143:6789/0 conn(0x7f08d80a98f0 legacy=0x7f08d8137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 1 monclient: found mon.b 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d81373e0 con 0x7f08d81430c0 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08dd006640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.928+0000 7f08dd006640 10 monclient: _renew_subs 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.929+0000 7f08dd006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:56.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.929+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d81358e0 con 0x7f08d81430c0 2024-03-25T19:30:56.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.929+0000 7f08dd006640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:56.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.929+0000 7f08dd006640 10 monclient: _renew_subs 2024-03-25T19:30:56.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.929+0000 7f08dd006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:56.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.929+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d8136150 con 0x7f08d81430c0 2024-03-25T19:30:56.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.929+0000 7f08dd006640 1 librados: init done 2024-03-25T19:30:56.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.930+0000 7f08dd006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:56.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.930+0000 7f08dd006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:56.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.930+0000 7f08dd006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:56.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:56.930+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d80627b0 con 0x7f08d81430c0 2024-03-25T19:30:57.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.210+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08d0001fb0 con 0x7f08d81430c0 2024-03-25T19:30:57.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.210+0000 7f08bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.210+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f08d0003b10 con 0x7f08d81430c0 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.210+0000 7f08bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.210+0000 7f08bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.210+0000 7f08bf7fe640 10 monclient: dump: 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:30:57.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.210+0000 7f08d559a640 4 set_mon_vals no callback set 2024-03-25T19:30:57.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.211+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f08d0003d30 con 0x7f08d81430c0 2024-03-25T19:30:57.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.212+0000 7f08d6d9d640 10 monclient: get_auth_request con 0x7f08c406dfb0 auth_method 0 2024-03-25T19:30:57.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.212+0000 7f08bf7fe640 10 client.4270.objecter ms_handle_connect 0x7f08c406dfb0 2024-03-25T19:30:57.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.217+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f08d0053780 con 0x7f08d81430c0 2024-03-25T19:30:57.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.217+0000 7f08bf7fe640 10 client.4270.objecter ms_dispatch 0x7f08d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:57.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.217+0000 7f08bf7fe640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:57.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.217+0000 7f08bf7fe640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:57.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.217+0000 7f08bf7fe640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T19:30:57.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.217+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08d004f290 con 0x7f08d81430c0 2024-03-25T19:30:57.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.217+0000 7f08bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:57.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.217+0000 7f08bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:57.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.453+0000 7f08dd006640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.453+0000 7f08dd006640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.453+0000 7f08dd006640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:30:57.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.453+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08d813e250 con 0x7f08d81430c0 2024-03-25T19:30:57.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.530+0000 7f08bf7fe640 1 -- 172.21.15.143:0/3284339354 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f08d00511c0 con 0x7f08d81430c0 2024-03-25T19:30:57.531 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.530+0000 7f08bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:57.531 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.530+0000 7f08bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:57.531 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:30:57.531 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 10 librados: watch_flush enter 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 10 librados: watch_flush exit 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 10 client.4270.objecter successfully canceled tick 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 >> v1:172.21.15.143:6800/1292282666 conn(0x7f08c406dfb0 legacy=0x7f08c4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 10 monclient: shutdown 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 >> v1:172.21.15.203:6789/0 conn(0x7f08d81430c0 legacy=0x7f08d81db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:57.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 shutdown_connections 2024-03-25T19:30:57.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 >> 172.21.15.143:0/3284339354 conn(0x7f08d80b7ea0 msgr2=0x7f08d814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:57.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 shutdown_connections 2024-03-25T19:30:57.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 1 -- 172.21.15.143:0/3284339354 wait complete. 2024-03-25T19:30:57.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:30:57.533+0000 7f08dd006640 1 librados: shutdown 2024-03-25T19:30:57.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:04.351 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:04.465 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.466+0000 7f4698433640 10 monclient: get_monmap_and_config 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.466+0000 7f4698433640 10 monclient: build_initial_monmap 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.466+0000 7f4698433640 10 monclient: monmap: 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:31:04.467003+0000 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:31:04.467003+0000 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:31:04.466 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:04.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.466+0000 7f4698433640 1 Processor -- start 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.466+0000 7f4698433640 1 -- start start 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.466+0000 7f4698433640 10 monclient: init 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 10 monclient(hunting): picked mon.noname-b con 0x7f46900a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 10 monclient(hunting): picked mon.noname-c con 0x7f469013f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 10 monclient(hunting): picked mon.noname-a con 0x7f46901430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4690142520 con 0x7f46901430c0 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46900b28a0 con 0x7f469013f710 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46900b2a30 con 0x7f46900a98f0 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:04.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.467+0000 7f4698433640 10 monclient(hunting): authenticate will time out at 6080.811446s 2024-03-25T19:31:04.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.471+0000 7f46961cf640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f46900a98f0 0x7f46900a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:49448/0 (socket says 172.21.15.143:49448) 2024-03-25T19:31:04.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.471+0000 7f46961cf640 1 -- 172.21.15.143:0/1826574700 learned_addr learned my addr 172.21.15.143:0/1826574700 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.490+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851275144 0 0) 0x7f46900b28a0 con 0x7f469013f710 2024-03-25T19:31:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.490+0000 7f46951cd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.490+0000 7f46951cd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.490+0000 7f46951cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.490+0000 7f46951cd640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:31:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.490+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f467c003520 con 0x7f469013f710 2024-03-25T19:31:04.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.494+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294406391 0 0) 0x7f46900b2a30 con 0x7f46900a98f0 2024-03-25T19:31:04.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.494+0000 7f46951cd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.494+0000 7f46951cd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.494+0000 7f46951cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.494+0000 7f46951cd640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:31:04.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.494+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46900b28a0 con 0x7f46900a98f0 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588677487 0 0) 0x7f467c003520 con 0x7f469013f710 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 >> v1:172.21.15.143:6790/0 conn(0x7f46900a98f0 legacy=0x7f46900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 >> v1:172.21.15.143:6789/0 conn(0x7f46901430c0 legacy=0x7f469014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 1 monclient: found mon.noname-c 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46900b44c0 con 0x7f469013f710 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 10 monclient: _finish_auth 0 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:04.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4680001690 con 0x7f469013f710 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f46951cd640 10 monclient: dump: 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:04.502 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:04.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f4698433640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:31:04.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.501+0000 7f4698433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.518+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46800017f0 con 0x7f469013f710 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.518+0000 7f46951cd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.518+0000 7f46951cd640 1 -- 172.21.15.143:0/1826574700 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4680001c00 con 0x7f469013f710 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.518+0000 7f4698433640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.518+0000 7f4698433640 4 set_mon_vals no callback set 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.518+0000 7f46951cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.518+0000 7f46951cd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.518+0000 7f46951cd640 10 monclient: dump: 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.518 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.519 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:04.519 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:04.519 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:04.519 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:04.519 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.519+0000 7f4698433640 10 monclient: shutdown 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.519+0000 7f4698433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.519+0000 7f4698433640 1 -- 172.21.15.143:0/1826574700 >> v1:172.21.15.203:6789/0 conn(0x7f469013f710 legacy=0x7f46901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.519+0000 7f4698433640 1 -- 172.21.15.143:0/1826574700 shutdown_connections 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.519+0000 7f4698433640 1 -- 172.21.15.143:0/1826574700 >> 172.21.15.143:0/1826574700 conn(0x7f46900b7ea0 msgr2=0x7f46900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.519+0000 7f4698433640 1 -- 172.21.15.143:0/1826574700 shutdown_connections 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.519+0000 7f4698433640 1 -- 172.21.15.143:0/1826574700 wait complete. 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.520+0000 7f4698433640 10 monclient: build_initial_monmap 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.520+0000 7f4698433640 10 monclient: monmap: 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:04.520 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:04.521 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:31:04.521 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:31:04.521 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:04.521 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:31:04.521 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:04.521 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:31:04.521 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:31:04.521 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:04.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.520+0000 7f4698433640 1 librados: starting msgr at 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.520+0000 7f4698433640 1 librados: starting objecter 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 1 Processor -- start 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 1 -- start start 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 1 librados: setting wanted keys 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 1 librados: calling monclient init 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 10 monclient: init 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 10 monclient(hunting): picked mon.noname-c con 0x7f46900a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 10 monclient(hunting): picked mon.noname-a con 0x7f469013f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 10 monclient(hunting): picked mon.noname-b con 0x7f46901430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4690139a40 con 0x7f469013f710 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.521+0000 7f4698433640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4690133f20 con 0x7f46901430c0 2024-03-25T19:31:04.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.522+0000 7f4698433640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46900ab950 con 0x7f46900a98f0 2024-03-25T19:31:04.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.522+0000 7f4698433640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:04.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.522+0000 7f4698433640 10 monclient(hunting): authenticate will time out at 6080.865708s 2024-03-25T19:31:04.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.528+0000 7f46961cf640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f46900a98f0 0x7f4690138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:49480/0 (socket says 172.21.15.143:49480) 2024-03-25T19:31:04.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.528+0000 7f46961cf640 1 -- 172.21.15.143:0/4142001070 learned_addr learned my addr 172.21.15.143:0/4142001070 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:04.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.530+0000 7f4686ffd640 10 client.?.objecter ms_handle_connect 0x7f46900a98f0 2024-03-25T19:31:04.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.530+0000 7f4686ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.535 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.536+0000 7f4686ffd640 10 client.?.objecter ms_handle_connect 0x7f46901430c0 2024-03-25T19:31:04.535 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.536+0000 7f4686ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.537+0000 7f4686ffd640 10 client.?.objecter ms_handle_connect 0x7f469013f710 2024-03-25T19:31:04.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.537+0000 7f4686ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:04.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.538+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562755179 0 0) 0x7f46900ab950 con 0x7f46900a98f0 2024-03-25T19:31:04.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.538+0000 7f4686ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.538+0000 7f4686ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.538+0000 7f4686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.538+0000 7f4686ffd640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:31:04.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.538+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4674004eb0 con 0x7f46900a98f0 2024-03-25T19:31:04.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.548+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220882159 0 0) 0x7f4690139a40 con 0x7f469013f710 2024-03-25T19:31:04.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.548+0000 7f4686ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.548+0000 7f4686ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.548+0000 7f4686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.548+0000 7f4686ffd640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:31:04.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.548+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46900ab950 con 0x7f469013f710 2024-03-25T19:31:04.550 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.550+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315497060 0 0) 0x7f4690133f20 con 0x7f46901430c0 2024-03-25T19:31:04.550 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.550+0000 7f4686ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:04.550 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.550+0000 7f4686ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:04.550 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.550+0000 7f4686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:04.550 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.550+0000 7f4686ffd640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:31:04.550 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.550+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4690139a40 con 0x7f46901430c0 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.553+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3152008021 0 0) 0x7f4674004eb0 con 0x7f46900a98f0 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.553+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4690133f20 con 0x7f46900a98f0 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.553+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f468c0019b0 con 0x7f46900a98f0 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.553+0000 7f4686ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.553+0000 7f4686ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.553+0000 7f4686ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:04.553 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:04.554 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:04.554 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:04.554 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.553+0000 7f4686ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:04.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.566+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048406553 0 0) 0x7f4690139a40 con 0x7f46901430c0 2024-03-25T19:31:04.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.566+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4674004eb0 con 0x7f46901430c0 2024-03-25T19:31:04.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.566+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4688001a60 con 0x7f46901430c0 2024-03-25T19:31:04.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.566+0000 7f4686ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.566+0000 7f4686ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.566+0000 7f4686ffd640 10 monclient(hunting): dump: 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:04.566 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2932428655 0 0) 0x7f4690133f20 con 0x7f46900a98f0 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 >> v1:172.21.15.203:6789/0 conn(0x7f46901430c0 legacy=0x7f4690133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 >> v1:172.21.15.143:6789/0 conn(0x7f469013f710 legacy=0x7f4690133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 1 monclient: found mon.c 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46900abc30 con 0x7f46900a98f0 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:04.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4686ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 5 monclient: authenticate success, global_id 4265 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 10 monclient: _renew_subs 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46900abf10 con 0x7f46900a98f0 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 10 monclient: _renew_subs 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4690139370 con 0x7f46900a98f0 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.567+0000 7f4698433640 1 librados: init done 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.568+0000 7f4698433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.568+0000 7f4698433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.568+0000 7f4698433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:04.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.568+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46900627b0 con 0x7f46900a98f0 2024-03-25T19:31:04.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.871+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f468c0047e0 con 0x7f46900a98f0 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.871+0000 7f4686ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.871+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f468c004b30 con 0x7f46900a98f0 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.871+0000 7f4686ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.871+0000 7f4686ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.871+0000 7f4686ffd640 10 monclient: dump: 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:04.871 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:04.872 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:04.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.871+0000 7f46949cc640 4 set_mon_vals no callback set 2024-03-25T19:31:04.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.872+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f468c004d50 con 0x7f46900a98f0 2024-03-25T19:31:04.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.872+0000 7f46959ce640 10 monclient: get_auth_request con 0x7f467406df70 auth_method 0 2024-03-25T19:31:04.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.873+0000 7f4686ffd640 10 client.4265.objecter ms_handle_connect 0x7f467406df70 2024-03-25T19:31:04.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.878+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f468c054890 con 0x7f46900a98f0 2024-03-25T19:31:04.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.878+0000 7f4686ffd640 10 client.4265.objecter ms_dispatch 0x7f4690000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:04.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.878+0000 7f4686ffd640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:04.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.878+0000 7f4686ffd640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:04.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.878+0000 7f4686ffd640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-25T19:31:04.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.878+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f468c0503a0 con 0x7f46900a98f0 2024-03-25T19:31:04.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.879+0000 7f4686ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:04.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:04.879+0000 7f4686ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:05.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.112+0000 7f4698433640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:05.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.112+0000 7f4698433640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:05.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.112+0000 7f4698433640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:05.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.112+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46900b7180 con 0x7f46900a98f0 2024-03-25T19:31:05.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.136+0000 7f4686ffd640 1 -- 172.21.15.143:0/4142001070 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f468c0522d0 con 0x7f46900a98f0 2024-03-25T19:31:05.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.136+0000 7f4686ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:05.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.136+0000 7f4686ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:05.136 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:31:05.136 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:05.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 10 librados: watch_flush enter 2024-03-25T19:31:05.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 10 librados: watch_flush exit 2024-03-25T19:31:05.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-25T19:31:05.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 10 client.4265.objecter successfully canceled tick 2024-03-25T19:31:05.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 >> v1:172.21.15.143:6800/1292282666 conn(0x7f467406df70 legacy=0x7f4674070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 10 monclient: shutdown 2024-03-25T19:31:05.140 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:05.140 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 >> v1:172.21.15.143:6790/0 conn(0x7f46900a98f0 legacy=0x7f4690138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:05.140 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 shutdown_connections 2024-03-25T19:31:05.140 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 >> 172.21.15.143:0/4142001070 conn(0x7f46900b7ea0 msgr2=0x7f4690074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:05.140 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 shutdown_connections 2024-03-25T19:31:05.140 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 1 -- 172.21.15.143:0/4142001070 wait complete. 2024-03-25T19:31:05.140 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:05.139+0000 7f4698433640 1 librados: shutdown 2024-03-25T19:31:05.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:11.956 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:12.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.071+0000 7f863744a640 10 monclient: get_monmap_and_config 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.071+0000 7f863744a640 10 monclient: build_initial_monmap 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.071+0000 7f863744a640 10 monclient: monmap: 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:31:12.072246+0000 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:31:12.072246+0000 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:31:12.071 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:12.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.072+0000 7f863744a640 1 Processor -- start 2024-03-25T19:31:12.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.072+0000 7f863744a640 1 -- start start 2024-03-25T19:31:12.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.072+0000 7f863744a640 10 monclient: init 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 10 monclient(hunting): picked mon.noname-c con 0x7f86300a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 10 monclient(hunting): picked mon.noname-a con 0x7f863013f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 10 monclient(hunting): picked mon.noname-b con 0x7f86301430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630142520 con 0x7f863013f710 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300b28a0 con 0x7f86300a98f0 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86300b2a30 con 0x7f86301430c0 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:12.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.073+0000 7f863744a640 10 monclient(hunting): authenticate will time out at 6088.417114s 2024-03-25T19:31:12.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.083+0000 7f86349e5640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f863013f710 0x7f86301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:49366/0 (socket says 172.21.15.143:49366) 2024-03-25T19:31:12.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.083+0000 7f86349e5640 1 -- 172.21.15.143:0/3124185398 learned_addr learned my addr 172.21.15.143:0/3124185398 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:12.095 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.095+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370985954 0 0) 0x7f8630142520 con 0x7f863013f710 2024-03-25T19:31:12.095 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.095+0000 7f8627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.095 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.095+0000 7f8627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.095 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.095+0000 7f8627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.095 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.095+0000 7f8627fff640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:31:12.095 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.095+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c003520 con 0x7f863013f710 2024-03-25T19:31:12.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.098+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251837545 0 0) 0x7f86300b28a0 con 0x7f86300a98f0 2024-03-25T19:31:12.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.098+0000 7f8627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.098+0000 7f8627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.098 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.098+0000 7f8627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.098 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.098+0000 7f8627fff640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:31:12.098 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.098+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630142520 con 0x7f86300a98f0 2024-03-25T19:31:12.104 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.104+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564076011 0 0) 0x7f86300b2a30 con 0x7f86301430c0 2024-03-25T19:31:12.104 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.105+0000 7f8627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.104 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.105+0000 7f8627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.104 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.105+0000 7f8627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.104 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.105+0000 7f8627fff640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:31:12.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.105+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86300b28a0 con 0x7f86301430c0 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2624832110 0 0) 0x7f861c003520 con 0x7f863013f710 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 >> v1:172.21.15.143:6790/0 conn(0x7f86301430c0 legacy=0x7f863014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 >> v1:172.21.15.203:6789/0 conn(0x7f86300a98f0 legacy=0x7f86300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 1 monclient: found mon.noname-a 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86300b44c0 con 0x7f863013f710 2024-03-25T19:31:12.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8620001690 con 0x7f863013f710 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f863744a640 5 monclient: authenticate success, global_id 4275 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f863744a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f8627fff640 10 monclient: dump: 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:12.108 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:12.109 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:12.109 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:12.109 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:12.109 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.107+0000 7f863744a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:12.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.111+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86200017f0 con 0x7f863013f710 2024-03-25T19:31:12.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.111+0000 7f8627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:12.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.111+0000 7f8627fff640 1 -- 172.21.15.143:0/3124185398 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8620001c00 con 0x7f863013f710 2024-03-25T19:31:12.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.111+0000 7f863744a640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:12.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.111+0000 7f8627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.111+0000 7f863744a640 4 set_mon_vals no callback set 2024-03-25T19:31:12.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.111+0000 7f8627fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:12.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.111+0000 7f8627fff640 10 monclient: dump: 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:12.112 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:12.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.112+0000 7f863744a640 10 monclient: shutdown 2024-03-25T19:31:12.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.112+0000 7f863744a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:12.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.112+0000 7f863744a640 1 -- 172.21.15.143:0/3124185398 >> v1:172.21.15.143:6789/0 conn(0x7f863013f710 legacy=0x7f86301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.112+0000 7f863744a640 1 -- 172.21.15.143:0/3124185398 shutdown_connections 2024-03-25T19:31:12.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.112+0000 7f863744a640 1 -- 172.21.15.143:0/3124185398 >> 172.21.15.143:0/3124185398 conn(0x7f86300b7ea0 msgr2=0x7f86300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:12.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.112+0000 7f863744a640 1 -- 172.21.15.143:0/3124185398 shutdown_connections 2024-03-25T19:31:12.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.113+0000 7f863744a640 1 -- 172.21.15.143:0/3124185398 wait complete. 2024-03-25T19:31:12.115 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.114+0000 7f863744a640 10 monclient: build_initial_monmap 2024-03-25T19:31:12.115 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.114+0000 7f863744a640 10 monclient: monmap: 2024-03-25T19:31:12.115 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:12.115 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:12.115 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:31:12.115 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.114+0000 7f863744a640 1 librados: starting msgr at 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.114+0000 7f863744a640 1 librados: starting objecter 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.114+0000 7f863744a640 1 Processor -- start 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.114+0000 7f863744a640 1 -- start start 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 1 librados: setting wanted keys 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 1 librados: calling monclient init 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 10 monclient: init 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:12.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:12.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 10 monclient(hunting): picked mon.noname-c con 0x7f86300a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:12.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 10 monclient(hunting): picked mon.noname-b con 0x7f863013f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:12.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 10 monclient(hunting): picked mon.noname-a con 0x7f86301430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:12.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630137f10 con 0x7f86301430c0 2024-03-25T19:31:12.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86301dc070 con 0x7f863013f710 2024-03-25T19:31:12.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8630137100 con 0x7f86300a98f0 2024-03-25T19:31:12.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:12.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.115+0000 7f863744a640 10 monclient(hunting): authenticate will time out at 6088.459564s 2024-03-25T19:31:12.125 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.126+0000 7f86359e7640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f86301430c0 0x7f86301db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:49412/0 (socket says 172.21.15.143:49412) 2024-03-25T19:31:12.126 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.126+0000 7f86359e7640 1 -- 172.21.15.143:0/2021103330 learned_addr learned my addr 172.21.15.143:0/2021103330 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:12.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.128+0000 7f8625ffb640 10 client.?.objecter ms_handle_connect 0x7f86301430c0 2024-03-25T19:31:12.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.128+0000 7f8625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:12.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.131+0000 7f8625ffb640 10 client.?.objecter ms_handle_connect 0x7f863013f710 2024-03-25T19:31:12.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.131+0000 7f8625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:12.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.134+0000 7f8625ffb640 10 client.?.objecter ms_handle_connect 0x7f86300a98f0 2024-03-25T19:31:12.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.134+0000 7f8625ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:12.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.144+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075663570 0 0) 0x7f8630137f10 con 0x7f86301430c0 2024-03-25T19:31:12.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.144+0000 7f8625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.144+0000 7f8625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.144 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.144+0000 7f8625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.144 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.144+0000 7f8625ffb640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:31:12.144 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.144+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8614004eb0 con 0x7f86301430c0 2024-03-25T19:31:12.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.146+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840104444 0 0) 0x7f8630137100 con 0x7f86300a98f0 2024-03-25T19:31:12.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.146+0000 7f8625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.146+0000 7f8625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.146+0000 7f8625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.146+0000 7f8625ffb640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:31:12.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.146+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630137f10 con 0x7f86300a98f0 2024-03-25T19:31:12.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.148+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917400779 0 0) 0x7f86301dc070 con 0x7f863013f710 2024-03-25T19:31:12.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.148+0000 7f8625ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:12.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.148+0000 7f8625ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:12.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.148+0000 7f8625ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:12.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.148+0000 7f8625ffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:31:12.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.148+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8630137100 con 0x7f863013f710 2024-03-25T19:31:12.156 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.157+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2469927709 0 0) 0x7f8614004eb0 con 0x7f86301430c0 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.157+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86301dc070 con 0x7f86301430c0 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.157+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8628001c70 con 0x7f86301430c0 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.157+0000 7f8625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.157+0000 7f8625ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.157+0000 7f8625ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:12.157 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:12.158 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.157+0000 7f8625ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:12.160 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.160+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2347295894 0 0) 0x7f8630137100 con 0x7f863013f710 2024-03-25T19:31:12.160 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.160+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8614004eb0 con 0x7f863013f710 2024-03-25T19:31:12.160 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.160+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8620001b00 con 0x7f863013f710 2024-03-25T19:31:12.160 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.160+0000 7f8625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.160 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.160+0000 7f8625ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:12.160 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.160+0000 7f8625ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:12.161 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:12.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.163+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2983750380 0 0) 0x7f8630137f10 con 0x7f86300a98f0 2024-03-25T19:31:12.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.163+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8630137100 con 0x7f86300a98f0 2024-03-25T19:31:12.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.163+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f862c001970 con 0x7f86300a98f0 2024-03-25T19:31:12.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.163+0000 7f8625ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.163+0000 7f8625ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.163+0000 7f8625ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:12.163 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:12.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1437087127 0 0) 0x7f8614004eb0 con 0x7f863013f710 2024-03-25T19:31:12.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 >> v1:172.21.15.143:6790/0 conn(0x7f86300a98f0 legacy=0x7f8630137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 >> v1:172.21.15.143:6789/0 conn(0x7f86301430c0 legacy=0x7f86301db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:12.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 1 monclient: found mon.b 2024-03-25T19:31:12.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:12.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86301373e0 con 0x7f863013f710 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f8625ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f863744a640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f863744a640 10 monclient: _renew_subs 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f863744a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86301358e0 con 0x7f863013f710 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f863744a640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.168+0000 7f863744a640 10 monclient: _renew_subs 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.169+0000 7f863744a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.169+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8630136150 con 0x7f863013f710 2024-03-25T19:31:12.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.169+0000 7f863744a640 1 librados: init done 2024-03-25T19:31:12.170 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.170+0000 7f863744a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:12.170 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.170+0000 7f863744a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:12.170 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.170+0000 7f863744a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:12.170 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.170+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86300627b0 con 0x7f863013f710 2024-03-25T19:31:12.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.453+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8620001c60 con 0x7f863013f710 2024-03-25T19:31:12.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.453+0000 7f8625ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:12.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.453+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8620004c30 con 0x7f863013f710 2024-03-25T19:31:12.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.453+0000 7f8625ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:12.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.453+0000 7f8625ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:12.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.453+0000 7f8625ffb640 10 monclient: dump: 2024-03-25T19:31:12.453 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:12.453 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:12.454 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.453+0000 7f86277fe640 4 set_mon_vals no callback set 2024-03-25T19:31:12.455 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.454+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f862001ac00 con 0x7f863013f710 2024-03-25T19:31:12.455 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.455+0000 7f86351e6640 10 monclient: get_auth_request con 0x7f861406e0b0 auth_method 0 2024-03-25T19:31:12.455 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.455+0000 7f8625ffb640 10 client.4282.objecter ms_handle_connect 0x7f861406e0b0 2024-03-25T19:31:12.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.459+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f8620054840 con 0x7f863013f710 2024-03-25T19:31:12.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.459+0000 7f8625ffb640 10 client.4282.objecter ms_dispatch 0x7f8630000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:12.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.459+0000 7f8625ffb640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:12.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.459+0000 7f8625ffb640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:12.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.459+0000 7f8625ffb640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T19:31:12.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.459+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8620050350 con 0x7f863013f710 2024-03-25T19:31:12.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.460+0000 7f8625ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:12.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.460+0000 7f8625ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:12.688 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:39.253 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.699 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:39.264 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.704+0000 7f863744a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:12.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.704+0000 7f863744a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:12.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.704+0000 7f863744a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:12.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.704+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8630135a70 con 0x7f863013f710 2024-03-25T19:31:12.802 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:39.367 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.808 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:39.374 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.819 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:39.384 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.850+0000 7f8625ffb640 1 -- 172.21.15.143:0/2021103330 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f8620052280 con 0x7f863013f710 2024-03-25T19:31:12.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.850+0000 7f8625ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:12.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.850+0000 7f8625ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:12.850 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:31:12.850 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:12.851 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:39.387 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 10 librados: watch_flush enter 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 10 librados: watch_flush exit 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 10 client.4282.objecter successfully canceled tick 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 >> v1:172.21.15.143:6800/1292282666 conn(0x7f861406e0b0 legacy=0x7f8614070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 10 monclient: shutdown 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 >> v1:172.21.15.203:6789/0 conn(0x7f863013f710 legacy=0x7f86301db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 shutdown_connections 2024-03-25T19:31:12.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 >> 172.21.15.143:0/2021103330 conn(0x7f86300b7ea0 msgr2=0x7f863014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:12.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 shutdown_connections 2024-03-25T19:31:12.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 1 -- 172.21.15.143:0/2021103330 wait complete. 2024-03-25T19:31:12.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:12.853+0000 7f863744a640 1 librados: shutdown 2024-03-25T19:31:12.858 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:39.394 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:12.867 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:39.432 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:12.898 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:39.434 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:19.670 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:19.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.787+0000 7f17ffa0f640 10 monclient: get_monmap_and_config 2024-03-25T19:31:19.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.787+0000 7f17ffa0f640 10 monclient: build_initial_monmap 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.787+0000 7f17ffa0f640 10 monclient: monmap: 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:31:19.788178+0000 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:31:19.788178+0000 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:31:19.787 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.787+0000 7f17ffa0f640 1 Processor -- start 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 1 -- start start 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 10 monclient: init 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f17f80b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f17f80b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f17f814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f80af940 con 0x7f17f80b4d60 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f80b0a70 con 0x7f17f80b1730 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f80b0d50 con 0x7f17f814be90 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:19.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.788+0000 7f17ffa0f640 10 monclient(hunting): authenticate will time out at 6096.132496s 2024-03-25T19:31:19.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.806+0000 7f17fcfaa640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f17f80b1730 0x7f17f80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:36200/0 (socket says 172.21.15.143:36200) 2024-03-25T19:31:19.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.806+0000 7f17fcfaa640 1 -- 172.21.15.143:0/3013281190 learned_addr learned my addr 172.21.15.143:0/3013281190 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:19.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.820+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477553428 0 0) 0x7f17f80b0a70 con 0x7f17f80b1730 2024-03-25T19:31:19.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.820+0000 7f17effff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:19.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.820+0000 7f17effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:19.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.820+0000 7f17effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:19.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.820+0000 7f17effff640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:31:19.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.820+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17e4003520 con 0x7f17f80b1730 2024-03-25T19:31:19.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.826+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738069827 0 0) 0x7f17f80af940 con 0x7f17f80b4d60 2024-03-25T19:31:19.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.826+0000 7f17effff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:19.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.826+0000 7f17effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:19.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.826+0000 7f17effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:19.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.826+0000 7f17effff640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:31:19.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.826+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f80b0a70 con 0x7f17f80b4d60 2024-03-25T19:31:19.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.831+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460426850 0 0) 0x7f17f80b0d50 con 0x7f17f814be90 2024-03-25T19:31:19.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.832+0000 7f17effff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:19.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.832+0000 7f17effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:19.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.832+0000 7f17effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:19.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.832+0000 7f17effff640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:31:19.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.832+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f80af940 con 0x7f17f814be90 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2769185267 0 0) 0x7f17e4003520 con 0x7f17f80b1730 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 >> v1:172.21.15.143:6790/0 conn(0x7f17f814be90 legacy=0x7f17f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 >> v1:172.21.15.143:6789/0 conn(0x7f17f80b4d60 legacy=0x7f17f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 1 monclient: found mon.noname-c 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f80b1030 con 0x7f17f80b1730 2024-03-25T19:31:19.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 10 monclient: _finish_auth 0 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f17e8001690 con 0x7f17f80b1730 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17effff640 10 monclient: dump: 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:19.844 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:19.845 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:19.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17ffa0f640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:31:19.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.843+0000 7f17ffa0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.861+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17e80017f0 con 0x7f17f80b1730 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.861+0000 7f17effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.861+0000 7f17effff640 1 -- 172.21.15.143:0/3013281190 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f17e8001c00 con 0x7f17f80b1730 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.861+0000 7f17effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.861+0000 7f17effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.861+0000 7f17effff640 10 monclient: dump: 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:19.861 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:19.862 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:19.862 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:19.862 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:19.862 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:19.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.861+0000 7f17ffa0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:19.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.861+0000 7f17ffa0f640 4 set_mon_vals no callback set 2024-03-25T19:31:19.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.862+0000 7f17ffa0f640 10 monclient: shutdown 2024-03-25T19:31:19.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.862+0000 7f17ffa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:19.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.862+0000 7f17ffa0f640 1 -- 172.21.15.143:0/3013281190 >> v1:172.21.15.203:6789/0 conn(0x7f17f80b1730 legacy=0x7f17f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:19.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.862+0000 7f17ffa0f640 1 -- 172.21.15.143:0/3013281190 shutdown_connections 2024-03-25T19:31:19.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.862+0000 7f17ffa0f640 1 -- 172.21.15.143:0/3013281190 >> 172.21.15.143:0/3013281190 conn(0x7f17f80617a0 msgr2=0x7f17f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:19.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.862+0000 7f17ffa0f640 1 -- 172.21.15.143:0/3013281190 shutdown_connections 2024-03-25T19:31:19.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.862+0000 7f17ffa0f640 1 -- 172.21.15.143:0/3013281190 wait complete. 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.863+0000 7f17ffa0f640 10 monclient: build_initial_monmap 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.863+0000 7f17ffa0f640 10 monclient: monmap: 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.863+0000 7f17ffa0f640 1 librados: starting msgr at 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.863+0000 7f17ffa0f640 1 librados: starting objecter 2024-03-25T19:31:19.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 1 Processor -- start 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 1 -- start start 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 1 librados: setting wanted keys 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 1 librados: calling monclient init 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 10 monclient: init 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f17f80b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f17f80b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f17f814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f8142050 con 0x7f17f814be90 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f81dac50 con 0x7f17f80b4d60 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f8140d00 con 0x7f17f80b1730 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:19.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.864+0000 7f17ffa0f640 10 monclient(hunting): authenticate will time out at 6096.208534s 2024-03-25T19:31:19.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.874+0000 7f17fcfaa640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f17f80b4d60 0x7f17f81db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:36246/0 (socket says 172.21.15.143:36246) 2024-03-25T19:31:19.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.874+0000 7f17fcfaa640 1 -- 172.21.15.143:0/1749515028 learned_addr learned my addr 172.21.15.143:0/1749515028 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:19.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.876+0000 7f17edffb640 10 client.?.objecter ms_handle_connect 0x7f17f80b4d60 2024-03-25T19:31:19.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.876+0000 7f17edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:19.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.880+0000 7f17edffb640 10 client.?.objecter ms_handle_connect 0x7f17f80b1730 2024-03-25T19:31:19.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.880+0000 7f17edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:19.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.883+0000 7f17edffb640 10 client.?.objecter ms_handle_connect 0x7f17f814be90 2024-03-25T19:31:19.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.883+0000 7f17edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:19.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.889+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312894515 0 0) 0x7f17f8140d00 con 0x7f17f80b1730 2024-03-25T19:31:19.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.889+0000 7f17edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:19.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.889+0000 7f17edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:19.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.889+0000 7f17edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:19.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.889+0000 7f17edffb640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:31:19.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.889+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17dc004eb0 con 0x7f17f80b1730 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855992677 0 0) 0x7f17f8142050 con 0x7f17f814be90 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f8140d00 con 0x7f17f814be90 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076114589 0 0) 0x7f17f81dac50 con 0x7f17f80b4d60 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:31:19.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.893+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f8142050 con 0x7f17f80b4d60 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.900+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 153944032 0 0) 0x7f17f8142050 con 0x7f17f80b4d60 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.900+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17f81dac50 con 0x7f17f80b4d60 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.900+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f17e8001b00 con 0x7f17f80b4d60 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.900+0000 7f17edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.900+0000 7f17edffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.900+0000 7f17edffb640 10 monclient(hunting): dump: 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.900 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:19.901 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:19.901 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:19.901 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:19.901 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:19.901 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:19.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.900+0000 7f17edffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.904+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 44457099 0 0) 0x7f17dc004eb0 con 0x7f17f80b1730 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.904+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17f8142050 con 0x7f17f80b1730 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.904+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f17f4001970 con 0x7f17f80b1730 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.904+0000 7f17edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.904+0000 7f17edffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.904+0000 7f17edffb640 10 monclient(hunting): dump: 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.904 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.905 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:19.905 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:19.905 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:19.905 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:19.905 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:19.905 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:19.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.917+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3076583337 0 0) 0x7f17f8140d00 con 0x7f17f814be90 2024-03-25T19:31:19.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.917+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17dc004eb0 con 0x7f17f814be90 2024-03-25T19:31:19.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.917+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f17f0001b50 con 0x7f17f814be90 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.917+0000 7f17edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.917+0000 7f17edffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.917+0000 7f17edffb640 10 monclient(hunting): dump: 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:19.917 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3808202093 0 0) 0x7f17f81dac50 con 0x7f17f80b4d60 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 >> v1:172.21.15.143:6790/0 conn(0x7f17f80b1730 legacy=0x7f17f81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 >> v1:172.21.15.143:6789/0 conn(0x7f17f814be90 legacy=0x7f17f81e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 1 monclient: found mon.b 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f81423e0 con 0x7f17f80b4d60 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17ffa0f640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:31:19.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17ffa0f640 10 monclient: _renew_subs 2024-03-25T19:31:19.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17ffa0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:19.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f813b470 con 0x7f17f80b4d60 2024-03-25T19:31:19.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17ffa0f640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:19.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17ffa0f640 10 monclient: _renew_subs 2024-03-25T19:31:19.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17ffa0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:19.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.918+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f81dba40 con 0x7f17f80b4d60 2024-03-25T19:31:19.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.919+0000 7f17ffa0f640 1 librados: init done 2024-03-25T19:31:19.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.920+0000 7f17ffa0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:19.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.920+0000 7f17ffa0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:19.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.920+0000 7f17ffa0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:19.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:19.920+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f81401f0 con 0x7f17f80b4d60 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.215+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17e8001230 con 0x7f17f80b4d60 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.215+0000 7f17edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.215+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f17e8004c30 con 0x7f17f80b4d60 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.215+0000 7f17edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.215+0000 7f17edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.215+0000 7f17edffb640 10 monclient: dump: 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:20.215 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:20.216 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:20.216 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:20.216 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:20.216 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:20.216 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:20.216 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:20.216 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:20.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.215+0000 7f17ef7fe640 4 set_mon_vals no callback set 2024-03-25T19:31:20.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.216+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f17e801ac40 con 0x7f17f80b4d60 2024-03-25T19:31:20.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.216+0000 7f17fd7ab640 10 monclient: get_auth_request con 0x7f17dc06e0f0 auth_method 0 2024-03-25T19:31:20.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.216+0000 7f17edffb640 10 client.4288.objecter ms_handle_connect 0x7f17dc06e0f0 2024-03-25T19:31:20.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.222+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f17e8054820 con 0x7f17f80b4d60 2024-03-25T19:31:20.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.222+0000 7f17edffb640 10 client.4288.objecter ms_dispatch 0x7f17f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:20.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.222+0000 7f17edffb640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:20.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.222+0000 7f17edffb640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:20.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.222+0000 7f17edffb640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:31:20.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.224+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17e8050330 con 0x7f17f80b4d60 2024-03-25T19:31:20.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.224+0000 7f17edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:20.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.224+0000 7f17edffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:20.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.471+0000 7f17ffa0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:20.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.471+0000 7f17ffa0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:20.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.471+0000 7f17ffa0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:20.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.471+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17f8140e90 con 0x7f17f80b4d60 2024-03-25T19:31:20.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.495+0000 7f17edffb640 1 -- 172.21.15.143:0/1749515028 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f17e8052260 con 0x7f17f80b4d60 2024-03-25T19:31:20.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.495+0000 7f17edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:20.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.495+0000 7f17edffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:20.495 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:31:20.495 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.497+0000 7f17ffa0f640 10 librados: watch_flush enter 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.497+0000 7f17ffa0f640 10 librados: watch_flush exit 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.497+0000 7f17ffa0f640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.497+0000 7f17ffa0f640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 >> v1:172.21.15.143:6800/1292282666 conn(0x7f17dc06e0f0 legacy=0x7f17dc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 10 monclient: shutdown 2024-03-25T19:31:20.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:20.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 >> v1:172.21.15.203:6789/0 conn(0x7f17f80b4d60 legacy=0x7f17f81db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:20.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 shutdown_connections 2024-03-25T19:31:20.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 >> 172.21.15.143:0/1749515028 conn(0x7f17f80617a0 msgr2=0x7f17f80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:20.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 shutdown_connections 2024-03-25T19:31:20.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 1 -- 172.21.15.143:0/1749515028 wait complete. 2024-03-25T19:31:20.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:20.498+0000 7f17ffa0f640 1 librados: shutdown 2024-03-25T19:31:20.510 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:27.317 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:27.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.434+0000 7fa6b15a8640 10 monclient: get_monmap_and_config 2024-03-25T19:31:27.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.434+0000 7fa6b15a8640 10 monclient: build_initial_monmap 2024-03-25T19:31:27.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.434+0000 7fa6b15a8640 10 monclient: monmap: 2024-03-25T19:31:27.434 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:27.434 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:27.434 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:31:27.435708+0000 2024-03-25T19:31:27.434 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:31:27.435708+0000 2024-03-25T19:31:27.435 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:27.435 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:27.435 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:27.435 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:31:27.435 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:31:27.435 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.435+0000 7fa6b15a8640 1 Processor -- start 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.435+0000 7fa6b15a8640 1 -- start start 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.435+0000 7fa6b15a8640 10 monclient: init 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6ac0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6ac13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6ac1430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac142520 con 0x7fa6ac13f710 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0b28a0 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac0b2a30 con 0x7fa6ac1430c0 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:27.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.436+0000 7fa6b15a8640 10 monclient(hunting): authenticate will time out at 6103.780094s 2024-03-25T19:31:27.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.446+0000 7fa6ab7fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fa6ac1430c0 0x7fa6ac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:44294/0 (socket says 172.21.15.143:44294) 2024-03-25T19:31:27.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.446+0000 7fa6ab7fe640 1 -- 172.21.15.143:0/3670400253 learned_addr learned my addr 172.21.15.143:0/3670400253 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:27.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.461+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047851532 0 0) 0x7fa6ac0b2a30 con 0x7fa6ac1430c0 2024-03-25T19:31:27.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.461+0000 7fa6a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.461+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.461+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.461+0000 7fa6a9ffb640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:31:27.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.461+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa694003520 con 0x7fa6ac1430c0 2024-03-25T19:31:27.465 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.466+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752871881 0 0) 0x7fa6ac142520 con 0x7fa6ac13f710 2024-03-25T19:31:27.465 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.466+0000 7fa6a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.465 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.466+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.465 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.466+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.466+0000 7fa6a9ffb640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:31:27.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.466+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac0b2a30 con 0x7fa6ac13f710 2024-03-25T19:31:27.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060144770 0 0) 0x7fa6ac0b28a0 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:31:27.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac142520 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1242295929 0 0) 0x7fa694003520 con 0x7fa6ac1430c0 2024-03-25T19:31:27.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 >> v1:172.21.15.203:6789/0 conn(0x7fa6ac0a98f0 legacy=0x7fa6ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 >> v1:172.21.15.143:6789/0 conn(0x7fa6ac13f710 legacy=0x7fa6ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 1 monclient: found mon.noname-b 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0b44c0 con 0x7fa6ac1430c0 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6a4001a10 con 0x7fa6ac1430c0 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.475+0000 7fa6a9ffb640 10 monclient: dump: 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.477 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.478 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.478 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:27.478 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:27.478 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:27.478 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:27.478 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:27.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.476+0000 7fa6b15a8640 5 monclient: authenticate success, global_id 4280 2024-03-25T19:31:27.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.476+0000 7fa6b15a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:27.487 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.488+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6a4001b70 con 0x7fa6ac1430c0 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.488+0000 7fa6a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.488+0000 7fa6a9ffb640 1 -- 172.21.15.143:0/3670400253 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6a4001f80 con 0x7fa6ac1430c0 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.488+0000 7fa6b15a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.488+0000 7fa6b15a8640 4 set_mon_vals no callback set 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.488+0000 7fa6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.488+0000 7fa6a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.488+0000 7fa6a9ffb640 10 monclient: dump: 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.488 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:27.489 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:27.489 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:27.489 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:27.489 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.489+0000 7fa6b15a8640 10 monclient: shutdown 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.489+0000 7fa6b15a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.489+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3670400253 >> v1:172.21.15.143:6790/0 conn(0x7fa6ac1430c0 legacy=0x7fa6ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.489+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3670400253 shutdown_connections 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.489+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3670400253 >> 172.21.15.143:0/3670400253 conn(0x7fa6ac0b7ea0 msgr2=0x7fa6ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.489+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3670400253 shutdown_connections 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.489+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3670400253 wait complete. 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 10 monclient: build_initial_monmap 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 10 monclient: monmap: 2024-03-25T19:31:27.491 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 1 librados: starting msgr at 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 1 librados: starting objecter 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 1 Processor -- start 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 1 -- start start 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 1 librados: setting wanted keys 2024-03-25T19:31:27.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 1 librados: calling monclient init 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.490+0000 7fa6b15a8640 10 monclient: init 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6ac0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6ac13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6ac1430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac1d7500 con 0x7fa6ac13f710 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac1d77e0 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ac1d8380 con 0x7fa6ac1430c0 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:27.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.491+0000 7fa6b15a8640 10 monclient(hunting): authenticate will time out at 6103.835500s 2024-03-25T19:31:27.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.507+0000 7fa6aaffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa6ac0a98f0 0x7fa6ac137c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:51644/0 (socket says 172.21.15.143:51644) 2024-03-25T19:31:27.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.507+0000 7fa6aaffd640 1 -- 172.21.15.143:0/3987164224 learned_addr learned my addr 172.21.15.143:0/3987164224 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:27.508 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.508+0000 7fa6937fe640 10 client.?.objecter ms_handle_connect 0x7fa6ac0a98f0 2024-03-25T19:31:27.508 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.508+0000 7fa6937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:27.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.515+0000 7fa6937fe640 10 client.?.objecter ms_handle_connect 0x7fa6ac13f710 2024-03-25T19:31:27.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.515+0000 7fa6937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:27.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.517+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782472833 0 0) 0x7fa6ac1d77e0 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.517+0000 7fa6937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.517+0000 7fa6937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.517+0000 7fa6937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.517+0000 7fa6937fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:31:27.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.517+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698004eb0 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.517+0000 7fa6937fe640 10 client.?.objecter ms_handle_connect 0x7fa6ac1430c0 2024-03-25T19:31:27.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.517+0000 7fa6937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:27.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.526+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441106096 0 0) 0x7fa6ac1d8380 con 0x7fa6ac1430c0 2024-03-25T19:31:27.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.526+0000 7fa6937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.526+0000 7fa6937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.526+0000 7fa6937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.526+0000 7fa6937fe640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:31:27.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.526+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac1d77e0 con 0x7fa6ac1430c0 2024-03-25T19:31:27.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.533+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470457392 0 0) 0x7fa6ac1d7500 con 0x7fa6ac13f710 2024-03-25T19:31:27.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.533+0000 7fa6937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:27.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.533+0000 7fa6937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:27.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.533+0000 7fa6937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:27.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.533+0000 7fa6937fe640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:31:27.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.534+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6ac1d8380 con 0x7fa6ac13f710 2024-03-25T19:31:27.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.543+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 496307618 0 0) 0x7fa698004eb0 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.543+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6ac1d7500 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.543+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa69c001a10 con 0x7fa6ac0a98f0 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.543+0000 7fa6937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.543+0000 7fa6937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.543+0000 7fa6937fe640 10 monclient(hunting): dump: 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:27.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.543+0000 7fa6937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:27.549 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.549+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228773595 0 0) 0x7fa6ac1d8380 con 0x7fa6ac13f710 2024-03-25T19:31:27.549 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.549+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa698004eb0 con 0x7fa6ac13f710 2024-03-25T19:31:27.549 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.549+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6a0001930 con 0x7fa6ac13f710 2024-03-25T19:31:27.549 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.550+0000 7fa6937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.549 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.550+0000 7fa6937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:27.549 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.550+0000 7fa6937fe640 10 monclient(hunting): dump: 2024-03-25T19:31:27.549 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:27.550 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:27.554 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.555+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1609378497 0 0) 0x7fa6ac1d77e0 con 0x7fa6ac1430c0 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.555+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6ac1d8380 con 0x7fa6ac1430c0 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.555+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6a4001e80 con 0x7fa6ac1430c0 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.555+0000 7fa6937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.555+0000 7fa6937fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.555+0000 7fa6937fe640 10 monclient(hunting): dump: 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:27.555 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:27.560 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 764642672 0 0) 0x7fa698004eb0 con 0x7fa6ac13f710 2024-03-25T19:31:27.560 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 >> v1:172.21.15.143:6790/0 conn(0x7fa6ac1430c0 legacy=0x7fa6ac133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.560 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 >> v1:172.21.15.203:6789/0 conn(0x7fa6ac0a98f0 legacy=0x7fa6ac137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:27.560 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 1 monclient: found mon.a 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0abcc0 con 0x7fa6ac13f710 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 5 monclient: authenticate success, global_id 4290 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 10 monclient: _renew_subs 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ac1384d0 con 0x7fa6ac13f710 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 10 monclient: _renew_subs 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ac1350d0 con 0x7fa6ac13f710 2024-03-25T19:31:27.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.560+0000 7fa6b15a8640 1 librados: init done 2024-03-25T19:31:27.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.561+0000 7fa6b15a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:27.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.561+0000 7fa6b15a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:27.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.561+0000 7fa6b15a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:27.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.561+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ac0627b0 con 0x7fa6ac13f710 2024-03-25T19:31:27.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.584+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6a00035f0 con 0x7fa6ac13f710 2024-03-25T19:31:27.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.584+0000 7fa6937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:27.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.584+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6a0003900 con 0x7fa6ac13f710 2024-03-25T19:31:27.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.584+0000 7fa6937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.584+0000 7fa6937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.584+0000 7fa6937fe640 10 monclient: dump: 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:27.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.584+0000 7fa6a97fa640 4 set_mon_vals no callback set 2024-03-25T19:31:27.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.585+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fa6a00198d0 con 0x7fa6ac13f710 2024-03-25T19:31:27.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.585+0000 7fa6aaffd640 10 monclient: get_auth_request con 0x7fa69806e110 auth_method 0 2024-03-25T19:31:27.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.585+0000 7fa6937fe640 10 client.4290.objecter ms_handle_connect 0x7fa69806e110 2024-03-25T19:31:27.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.588+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fa6a0053530 con 0x7fa6ac13f710 2024-03-25T19:31:27.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.588+0000 7fa6937fe640 10 client.4290.objecter ms_dispatch 0x7fa6ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:27.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.588+0000 7fa6937fe640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:27.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.588+0000 7fa6937fe640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:27.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.588+0000 7fa6937fe640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T19:31:27.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.863+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6a004f040 con 0x7fa6ac13f710 2024-03-25T19:31:27.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.864+0000 7fa6937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:27.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:27.864+0000 7fa6937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:28.077 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.077+0000 7fa6b15a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:28.077 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.077+0000 7fa6b15a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:28.077 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.077+0000 7fa6b15a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:28.077 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.077+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6ac135380 con 0x7fa6ac13f710 2024-03-25T19:31:28.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.110+0000 7fa6937fe640 1 -- 172.21.15.143:0/3987164224 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fa6a0050f70 con 0x7fa6ac13f710 2024-03-25T19:31:28.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.110+0000 7fa6937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:28.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.110+0000 7fa6937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:28.111 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:31:28.111 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:28.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.112+0000 7fa6b15a8640 10 librados: watch_flush enter 2024-03-25T19:31:28.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 10 librados: watch_flush exit 2024-03-25T19:31:28.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T19:31:28.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 10 client.4290.objecter successfully canceled tick 2024-03-25T19:31:28.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 >> v1:172.21.15.143:6800/1292282666 conn(0x7fa69806e110 legacy=0x7fa6980705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 10 monclient: shutdown 2024-03-25T19:31:28.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:28.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 >> v1:172.21.15.143:6789/0 conn(0x7fa6ac13f710 legacy=0x7fa6ac1d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:28.114 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 shutdown_connections 2024-03-25T19:31:28.114 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 >> 172.21.15.143:0/3987164224 conn(0x7fa6ac0b7ea0 msgr2=0x7fa6ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:28.114 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 shutdown_connections 2024-03-25T19:31:28.114 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 1 -- 172.21.15.143:0/3987164224 wait complete. 2024-03-25T19:31:28.114 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:28.113+0000 7fa6b15a8640 1 librados: shutdown 2024-03-25T19:31:28.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:32.949 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:59.514 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:32.958 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:59.523 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:33.042 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:01:59.607 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:33.072 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:59.608 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:33.076 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:59.611 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:33.086 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:59.651 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:33.093 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:59.658 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:33.111 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:01:59.647 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:33.129 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:01:59.694 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:34.931 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:35.044 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.044+0000 7f160fb0e640 10 monclient: get_monmap_and_config 2024-03-25T19:31:35.044 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.044+0000 7f160fb0e640 10 monclient: build_initial_monmap 2024-03-25T19:31:35.044 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.044+0000 7f160fb0e640 10 monclient: monmap: 2024-03-25T19:31:35.044 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:35.044 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:35.044 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:31:35.045359+0000 2024-03-25T19:31:35.045 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:31:35.045359+0000 2024-03-25T19:31:35.045 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:35.045 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:35.045 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:35.045 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:31:35.045 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:31:35.045 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:35.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.045+0000 7f160fb0e640 1 Processor -- start 2024-03-25T19:31:35.046 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.045+0000 7f160fb0e640 1 -- start start 2024-03-25T19:31:35.046 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.045+0000 7f160fb0e640 10 monclient: init 2024-03-25T19:31:35.046 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:35.046 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:35.046 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f16080a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:35.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f160813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:35.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f16081430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:35.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1608142520 con 0x7f16081430c0 2024-03-25T19:31:35.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16080b28a0 con 0x7f16080a98f0 2024-03-25T19:31:35.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16080b2a30 con 0x7f160813f710 2024-03-25T19:31:35.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:35.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.046+0000 7f160fb0e640 10 monclient(hunting): authenticate will time out at 6111.389851s 2024-03-25T19:31:35.050 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.051+0000 7f160d8aa640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f16080a98f0 0x7f16080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:56178/0 (socket says 172.21.15.143:56178) 2024-03-25T19:31:35.050 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.051+0000 7f160d8aa640 1 -- 172.21.15.143:0/2021775982 learned_addr learned my addr 172.21.15.143:0/2021775982 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:35.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.059+0000 7f160c8a8640 1 -- 172.21.15.143:0/2021775982 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110650348 0 0) 0x7f16080b28a0 con 0x7f16080a98f0 2024-03-25T19:31:35.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.059+0000 7f160c8a8640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.059+0000 7f160c8a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.059+0000 7f160c8a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.059+0000 7f160c8a8640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:31:35.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.059+0000 7f160c8a8640 1 -- 172.21.15.143:0/2021775982 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f4003520 con 0x7f16080a98f0 2024-03-25T19:31:35.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 1 -- 172.21.15.143:0/2021775982 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 590304295 0 0) 0x7f15f4003520 con 0x7f16080a98f0 2024-03-25T19:31:35.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 1 -- 172.21.15.143:0/2021775982 >> v1:172.21.15.143:6790/0 conn(0x7f160813f710 legacy=0x7f16081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 1 -- 172.21.15.143:0/2021775982 >> v1:172.21.15.143:6789/0 conn(0x7f16081430c0 legacy=0x7f160814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 10 monclient: _finish_hunting 0 2024-03-25T19:31:35.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 1 monclient: found mon.noname-c 2024-03-25T19:31:35.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:35.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:31:35.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 1 -- 172.21.15.143:0/2021775982 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16080b44c0 con 0x7f16080a98f0 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 10 monclient: _finish_auth 0 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 10 monclient: _check_auth_tickets 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 1 -- 172.21.15.143:0/2021775982 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1604001690 con 0x7f16080a98f0 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160c8a8640 10 monclient: dump: 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:35.067 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:35.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160fb0e640 5 monclient: authenticate success, global_id 4297 2024-03-25T19:31:35.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.066+0000 7f160fb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:35.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.070+0000 7f160c8a8640 1 -- 172.21.15.143:0/2021775982 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16040017f0 con 0x7f16080a98f0 2024-03-25T19:31:35.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.070+0000 7f160c8a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:35.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.070+0000 7f160fb0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:35.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.070+0000 7f160fb0e640 4 set_mon_vals no callback set 2024-03-25T19:31:35.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.071+0000 7f160fb0e640 10 monclient: shutdown 2024-03-25T19:31:35.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.071+0000 7f160fb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:35.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.071+0000 7f160fb0e640 1 -- 172.21.15.143:0/2021775982 >> v1:172.21.15.203:6789/0 conn(0x7f16080a98f0 legacy=0x7f16080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.071+0000 7f160fb0e640 1 -- 172.21.15.143:0/2021775982 shutdown_connections 2024-03-25T19:31:35.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.071+0000 7f160fb0e640 1 -- 172.21.15.143:0/2021775982 >> 172.21.15.143:0/2021775982 conn(0x7f16080b7ea0 msgr2=0x7f16080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:35.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.071+0000 7f160fb0e640 1 -- 172.21.15.143:0/2021775982 shutdown_connections 2024-03-25T19:31:35.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.071+0000 7f160fb0e640 1 -- 172.21.15.143:0/2021775982 wait complete. 2024-03-25T19:31:35.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.072+0000 7f160fb0e640 10 monclient: build_initial_monmap 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.072+0000 7f160fb0e640 10 monclient: monmap: 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:31:35.072 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.072+0000 7f160fb0e640 1 librados: starting msgr at 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.072+0000 7f160fb0e640 1 librados: starting objecter 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 1 Processor -- start 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 1 -- start start 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 1 librados: setting wanted keys 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 1 librados: calling monclient init 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 10 monclient: init 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f16080a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f160813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:35.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f16081430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:35.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1608138830 con 0x7f160813f710 2024-03-25T19:31:35.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16080b1700 con 0x7f16080a98f0 2024-03-25T19:31:35.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16080b1220 con 0x7f16081430c0 2024-03-25T19:31:35.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:35.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.073+0000 7f160fb0e640 10 monclient(hunting): authenticate will time out at 6111.417554s 2024-03-25T19:31:35.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.082+0000 7f160d0a9640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f160813f710 0x7f16081d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:50216/0 (socket says 172.21.15.143:50216) 2024-03-25T19:31:35.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.082+0000 7f160d0a9640 1 -- 172.21.15.143:0/39672047 learned_addr learned my addr 172.21.15.143:0/39672047 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:35.084 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.084+0000 7f15fe7fc640 10 client.?.objecter ms_handle_connect 0x7f160813f710 2024-03-25T19:31:35.084 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.084+0000 7f15fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:35.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.091+0000 7f15fe7fc640 10 client.?.objecter ms_handle_connect 0x7f16080a98f0 2024-03-25T19:31:35.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.091+0000 7f15fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:35.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.092+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928935583 0 0) 0x7f1608138830 con 0x7f160813f710 2024-03-25T19:31:35.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.092+0000 7f15fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.092+0000 7f15fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.092+0000 7f15fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.092+0000 7f15fe7fc640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:31:35.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.092+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ec004eb0 con 0x7f160813f710 2024-03-25T19:31:35.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.097+0000 7f15fe7fc640 10 client.?.objecter ms_handle_connect 0x7f16081430c0 2024-03-25T19:31:35.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.097+0000 7f15fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:35.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.101+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1422431594 0 0) 0x7f15ec004eb0 con 0x7f160813f710 2024-03-25T19:31:35.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.101+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1608138830 con 0x7f160813f710 2024-03-25T19:31:35.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.101+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f15f8001a30 con 0x7f160813f710 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.101+0000 7f15fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.101+0000 7f15fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.101+0000 7f15fe7fc640 10 monclient(hunting): dump: 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:35.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.101+0000 7f15fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:35.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.109+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784477796 0 0) 0x7f16080b1220 con 0x7f16081430c0 2024-03-25T19:31:35.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.109+0000 7f15fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:35.109 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.109+0000 7f15fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:35.109 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.109+0000 7f15fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:35.109 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.109+0000 7f15fe7fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:31:35.109 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.109+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ec004eb0 con 0x7f16081430c0 2024-03-25T19:31:35.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 505941352 0 0) 0x7f1608138830 con 0x7f160813f710 2024-03-25T19:31:35.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 >> v1:172.21.15.143:6790/0 conn(0x7f16081430c0 legacy=0x7f16081e1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 >> v1:172.21.15.203:6789/0 conn(0x7f16080a98f0 legacy=0x7f16081378c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:35.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 1 monclient: found mon.a 2024-03-25T19:31:35.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:35.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:35.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16080b13b0 con 0x7f160813f710 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f15fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 5 monclient: authenticate success, global_id 4296 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 10 monclient: _renew_subs 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16081d7b00 con 0x7f160813f710 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 10 monclient: _renew_subs 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16081e3590 con 0x7f160813f710 2024-03-25T19:31:35.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.110+0000 7f160fb0e640 1 librados: init done 2024-03-25T19:31:35.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.111+0000 7f160fb0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:35.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.111+0000 7f160fb0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:35.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.111+0000 7f160fb0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:35.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.111+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16081385d0 con 0x7f160813f710 2024-03-25T19:31:35.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.428+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15f8001cb0 con 0x7f160813f710 2024-03-25T19:31:35.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.428+0000 7f15fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.428+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f15f8003a60 con 0x7f160813f710 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.428+0000 7f15fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.428+0000 7f15fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.428+0000 7f15fe7fc640 10 monclient: dump: 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:35.428 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:35.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.428+0000 7f15fffff640 4 set_mon_vals no callback set 2024-03-25T19:31:35.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.428+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f15f8003c80 con 0x7f160813f710 2024-03-25T19:31:35.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.429+0000 7f160d8aa640 10 monclient: get_auth_request con 0x7f15ec06dd40 auth_method 0 2024-03-25T19:31:35.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.429+0000 7f15fe7fc640 10 client.4296.objecter ms_handle_connect 0x7f15ec06dd40 2024-03-25T19:31:35.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.435+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f15f80548d0 con 0x7f160813f710 2024-03-25T19:31:35.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.435+0000 7f15fe7fc640 10 client.4296.objecter ms_dispatch 0x7f1608000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:35.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.435+0000 7f15fe7fc640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:35.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.435+0000 7f15fe7fc640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:35.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.435+0000 7f15fe7fc640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-25T19:31:35.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.435+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15f8050210 con 0x7f160813f710 2024-03-25T19:31:35.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.435+0000 7f15fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:35.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.435+0000 7f15fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:35.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.664+0000 7f160fb0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:35.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.664+0000 7f160fb0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:35.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.664+0000 7f160fb0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:31:35.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.664+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16081d7c90 con 0x7f160813f710 2024-03-25T19:31:35.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.689+0000 7f15fe7fc640 1 -- 172.21.15.143:0/39672047 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f15f8052310 con 0x7f160813f710 2024-03-25T19:31:35.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.689+0000 7f15fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:35.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.689+0000 7f15fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:35.689 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:31:35.689 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:35.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 10 librados: watch_flush enter 2024-03-25T19:31:35.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 10 librados: watch_flush exit 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 10 client.4296.objecter successfully canceled tick 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 >> v1:172.21.15.143:6800/1292282666 conn(0x7f15ec06dd40 legacy=0x7f15ec070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 10 monclient: shutdown 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 >> v1:172.21.15.143:6789/0 conn(0x7f160813f710 legacy=0x7f16081d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.691+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 shutdown_connections 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.692+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 >> 172.21.15.143:0/39672047 conn(0x7f16080b7ea0 msgr2=0x7f1608074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.692+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 shutdown_connections 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.692+0000 7f160fb0e640 1 -- 172.21.15.143:0/39672047 wait complete. 2024-03-25T19:31:35.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:35.692+0000 7f160fb0e640 1 librados: shutdown 2024-03-25T19:31:35.703 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:42.509 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:42.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.620+0000 7f1f2658e640 10 monclient: get_monmap_and_config 2024-03-25T19:31:42.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.620+0000 7f1f2658e640 10 monclient: build_initial_monmap 2024-03-25T19:31:42.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.620+0000 7f1f2658e640 10 monclient: monmap: 2024-03-25T19:31:42.620 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:42.620 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:42.621 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:31:42.621844+0000 2024-03-25T19:31:42.621 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:31:42.621844+0000 2024-03-25T19:31:42.621 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:42.621 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:42.621 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:42.621 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:31:42.621 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:31:42.621 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:42.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.621+0000 7f1f2658e640 1 Processor -- start 2024-03-25T19:31:42.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 1 -- start start 2024-03-25T19:31:42.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 10 monclient: init 2024-03-25T19:31:42.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:42.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:42.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f200b0070 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:42.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2013c170 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:42.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f2006fa90 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:42.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200b0450 con 0x7f1f200b0070 2024-03-25T19:31:42.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f20071440 con 0x7f1f2006fa90 2024-03-25T19:31:42.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200b0bf0 con 0x7f1f2013c170 2024-03-25T19:31:42.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:42.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.622+0000 7f1f2658e640 10 monclient(hunting): authenticate will time out at 6118.966408s 2024-03-25T19:31:42.632 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.633+0000 7f1f1ffff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1f200b0070 0x7f1f2013ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51554/0 (socket says 172.21.15.143:51554) 2024-03-25T19:31:42.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.633+0000 7f1f1ffff640 1 -- 172.21.15.143:0/1049734010 learned_addr learned my addr 172.21.15.143:0/1049734010 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:42.640 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.641+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823659281 0 0) 0x7f1f20071440 con 0x7f1f2006fa90 2024-03-25T19:31:42.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.641+0000 7f1f1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:42.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.641+0000 7f1f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:42.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.641+0000 7f1f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:42.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.641+0000 7f1f1f7fe640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:31:42.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.641+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f0c003520 con 0x7f1f2006fa90 2024-03-25T19:31:42.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.653+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777371325 0 0) 0x7f1f200b0450 con 0x7f1f200b0070 2024-03-25T19:31:42.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.653+0000 7f1f1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:42.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.653+0000 7f1f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:42.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.653+0000 7f1f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:42.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.653+0000 7f1f1f7fe640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:31:42.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.653+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f20071440 con 0x7f1f200b0070 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47023636 0 0) 0x7f1f0c003520 con 0x7f1f2006fa90 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 >> v1:172.21.15.143:6790/0 conn(0x7f1f2013c170 legacy=0x7f1f20141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 >> v1:172.21.15.143:6789/0 conn(0x7f1f200b0070 legacy=0x7f1f2013ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200716b0 con 0x7f1f2006fa90 2024-03-25T19:31:42.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1f18001a10 con 0x7f1f2006fa90 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f1f7fe640 10 monclient: dump: 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:42.659 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:42.660 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:42.660 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:42.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f2658e640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:31:42.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.658+0000 7f1f2658e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.666+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f18001b70 con 0x7f1f2006fa90 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.666+0000 7f1f1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.666+0000 7f1f1f7fe640 1 -- 172.21.15.143:0/1049734010 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1f18001f80 con 0x7f1f2006fa90 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.666+0000 7f1f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.666+0000 7f1f1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.666+0000 7f1f1f7fe640 10 monclient: dump: 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:42.666 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.666+0000 7f1f2658e640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:42.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.666+0000 7f1f2658e640 4 set_mon_vals no callback set 2024-03-25T19:31:42.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.667+0000 7f1f2658e640 10 monclient: shutdown 2024-03-25T19:31:42.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.667+0000 7f1f2658e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:42.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.667+0000 7f1f2658e640 1 -- 172.21.15.143:0/1049734010 >> v1:172.21.15.203:6789/0 conn(0x7f1f2006fa90 legacy=0x7f1f2014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.667+0000 7f1f2658e640 1 -- 172.21.15.143:0/1049734010 shutdown_connections 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.667+0000 7f1f2658e640 1 -- 172.21.15.143:0/1049734010 >> 172.21.15.143:0/1049734010 conn(0x7f1f200b0ef0 msgr2=0x7f1f200b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.667+0000 7f1f2658e640 1 -- 172.21.15.143:0/1049734010 shutdown_connections 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.667+0000 7f1f2658e640 1 -- 172.21.15.143:0/1049734010 wait complete. 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.668+0000 7f1f2658e640 10 monclient: build_initial_monmap 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.668+0000 7f1f2658e640 10 monclient: monmap: 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:42.669 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.669+0000 7f1f2658e640 1 librados: starting msgr at 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.669+0000 7f1f2658e640 1 librados: starting objecter 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.669+0000 7f1f2658e640 1 Processor -- start 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.669+0000 7f1f2658e640 1 -- start start 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.669+0000 7f1f2658e640 1 librados: setting wanted keys 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.669+0000 7f1f2658e640 1 librados: calling monclient init 2024-03-25T19:31:42.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.669+0000 7f1f2658e640 10 monclient: init 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.669+0000 7f1f2658e640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 10 monclient(hunting): picked mon.noname-c con 0x7f1f2006fa90 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 10 monclient(hunting): picked mon.noname-a con 0x7f1f200b0070 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2013c170 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200a9b80 con 0x7f1f200b0070 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f201d9630 con 0x7f1f2013c170 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f201d97c0 con 0x7f1f2006fa90 2024-03-25T19:31:42.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:42.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.670+0000 7f1f2658e640 10 monclient(hunting): authenticate will time out at 6119.013893s 2024-03-25T19:31:42.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.677+0000 7f1f17fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1f200b0070 0x7f1f200abe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51604/0 (socket says 172.21.15.143:51604) 2024-03-25T19:31:42.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.677+0000 7f1f17fff640 1 -- 172.21.15.143:0/2291596167 learned_addr learned my addr 172.21.15.143:0/2291596167 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:42.685 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.686+0000 7f1f1d7fa640 10 client.?.objecter ms_handle_connect 0x7f1f2006fa90 2024-03-25T19:31:42.685 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.686+0000 7f1f1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:42.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.686+0000 7f1f1d7fa640 10 client.?.objecter ms_handle_connect 0x7f1f2013c170 2024-03-25T19:31:42.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.686+0000 7f1f1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:42.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.687+0000 7f1f1d7fa640 10 client.?.objecter ms_handle_connect 0x7f1f200b0070 2024-03-25T19:31:42.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.687+0000 7f1f1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:42.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.693+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249649548 0 0) 0x7f1f200a9b80 con 0x7f1f200b0070 2024-03-25T19:31:42.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.693+0000 7f1f1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:42.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.693+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:42.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.693+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:42.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.693+0000 7f1f1d7fa640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:31:42.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.693+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f00004eb0 con 0x7f1f200b0070 2024-03-25T19:31:42.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.695+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949585608 0 0) 0x7f1f201d97c0 con 0x7f1f2006fa90 2024-03-25T19:31:42.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.695+0000 7f1f1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:42.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.695+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:42.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.695+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:42.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.695+0000 7f1f1d7fa640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:31:42.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.695+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f200a9b80 con 0x7f1f2006fa90 2024-03-25T19:31:42.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.699+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807741817 0 0) 0x7f1f201d9630 con 0x7f1f2013c170 2024-03-25T19:31:42.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.699+0000 7f1f1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:42.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.699+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:42.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.699+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:42.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.699+0000 7f1f1d7fa640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:31:42.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.699+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f201d97c0 con 0x7f1f2013c170 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.705+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 124686453 0 0) 0x7f1f200a9b80 con 0x7f1f2006fa90 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.705+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f201d9630 con 0x7f1f2006fa90 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.705+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1f10001930 con 0x7f1f2006fa90 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.705+0000 7f1f1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.705+0000 7f1f1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.705+0000 7f1f1d7fa640 10 monclient(hunting): dump: 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:42.705 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:42.706 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:42.706 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:42.706 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:42.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.705+0000 7f1f1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:42.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1559639682 0 0) 0x7f1f201d9630 con 0x7f1f2006fa90 2024-03-25T19:31:42.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.203:6789/0 conn(0x7f1f2013c170 legacy=0x7f1f200ac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6789/0 conn(0x7f1f200b0070 legacy=0x7f1f200abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 1 monclient: found mon.c 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f201d9950 con 0x7f1f2006fa90 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 5 monclient: authenticate success, global_id 4295 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 10 monclient: _renew_subs 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2014a600 con 0x7f1f2006fa90 2024-03-25T19:31:42.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:42.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 10 monclient: _renew_subs 2024-03-25T19:31:42.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:42.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f201d8cf0 con 0x7f1f2006fa90 2024-03-25T19:31:42.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.710+0000 7f1f2658e640 1 librados: init done 2024-03-25T19:31:42.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.711+0000 7f1f2658e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:42.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.711+0000 7f1f2658e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:42.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.711+0000 7f1f2658e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:42.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:42.711+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f200627b0 con 0x7f1f2006fa90 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.009+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f10001d40 con 0x7f1f2006fa90 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.009+0000 7f1f1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.009+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1f10004980 con 0x7f1f2006fa90 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.009+0000 7f1f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.009+0000 7f1f1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.009+0000 7f1f1d7fa640 10 monclient: dump: 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:43.009 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:43.010 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:43.010 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.010 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:43.010 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:43.010 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:43.010 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:43.010 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:43.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.009+0000 7f1f1effd640 4 set_mon_vals no callback set 2024-03-25T19:31:43.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.010+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f1f10004ba0 con 0x7f1f2006fa90 2024-03-25T19:31:43.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.011+0000 7f1f17fff640 10 monclient: get_auth_request con 0x7f1f0006de20 auth_method 0 2024-03-25T19:31:43.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.011+0000 7f1f1d7fa640 10 client.4295.objecter ms_handle_connect 0x7f1f0006de20 2024-03-25T19:31:43.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.016+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f1f10054780 con 0x7f1f2006fa90 2024-03-25T19:31:43.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.016+0000 7f1f1d7fa640 10 client.4295.objecter ms_dispatch 0x7f1f20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:43.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.016+0000 7f1f1d7fa640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:43.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.016+0000 7f1f1d7fa640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:43.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.016+0000 7f1f1d7fa640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-25T19:31:43.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.017+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f100500c0 con 0x7f1f2006fa90 2024-03-25T19:31:43.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.017+0000 7f1f1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:43.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.017+0000 7f1f1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:43.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.273+0000 7f1f2658e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:43.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.273+0000 7f1f2658e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:43.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.273+0000 7f1f2658e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:43.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.273+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f2014c680 con 0x7f1f2006fa90 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.285+0000 7f1f1ffff640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6790/0 conn(0x7f1f2006fa90 legacy=0x7f1f200a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.285+0000 7f1f1ffff640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6790/0 conn(0x7f1f2006fa90 legacy=0x7f1f200a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.285+0000 7f1f1ffff640 1 --1- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6790/0 conn(0x7f1f2006fa90 0x7f1f200a9470 :-1 s=OPENED pgs=8554 cs=1 l=1).handle_message read tag failed 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.285+0000 7f1f1ffff640 1 --1- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6790/0 conn(0x7f1f2006fa90 0x7f1f200a9470 :-1 s=OPENED pgs=8554 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.143:6790/0 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6790/0 conn(0x7f1f2006fa90 legacy=0x7f1f200a9470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:43.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 10 monclient(hunting): picked mon.b con 0x7f1f2013c170 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 10 monclient(hunting): picked mon.a con 0x7f1f200b0070 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 10 monclient(hunting): picked mon.c con 0x7f1f0007a3a0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f201d9630 con 0x7f1f200b0070 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f200a9b80 con 0x7f1f2013c170 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f000804e0 con 0x7f1f0007a3a0 2024-03-25T19:31:43.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.286+0000 7f1f1d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:43.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.292+0000 7f1f1d7fa640 10 client.4295.objecter ms_handle_connect 0x7f1f2013c170 2024-03-25T19:31:43.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.292+0000 7f1f1d7fa640 10 client.4295.objecter resend_mon_ops 2024-03-25T19:31:43.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.298+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439827327 0 0) 0x7f1f200a9b80 con 0x7f1f2013c170 2024-03-25T19:31:43.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.298+0000 7f1f1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.298+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.298+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:31:43.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.298+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.203:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1f0007f4b0 con 0x7f1f2013c170 2024-03-25T19:31:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.307+0000 7f1f1d7fa640 10 client.4295.objecter ms_handle_connect 0x7f1f200b0070 2024-03-25T19:31:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.307+0000 7f1f1d7fa640 10 client.4295.objecter resend_mon_ops 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 3784734312 0 0) 0x7f1f0007f4b0 con 0x7f1f2013c170 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6790/0 conn(0x7f1f0007a3a0 legacy=0x7f1f0007c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6789/0 conn(0x7f1f200b0070 legacy=0x7f1f00078830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 1 monclient: found mon.b 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f1f00081740 con 0x7f1f2013c170 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f000804e0 con 0x7f1f2013c170 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1f100581d0 con 0x7f1f2013c170 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.308+0000 7f1f1d7fa640 10 monclient: dump: 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:43.309 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:43.310 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:43.310 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:43.310 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.310+0000 7f1f1ffff640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.203:6789/0 conn(0x7f1f2013c170 legacy=0x7f1f00074dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.310+0000 7f1f1ffff640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.203:6789/0 conn(0x7f1f2013c170 legacy=0x7f1f00074dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.310+0000 7f1f1ffff640 1 --1- 172.21.15.143:0/2291596167 >> v1:172.21.15.203:6789/0 conn(0x7f1f2013c170 0x7f1f00074dd0 :-1 s=OPENED pgs=8618 cs=1 l=1).handle_message read tag failed 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.310+0000 7f1f1ffff640 1 --1- 172.21.15.143:0/2291596167 >> v1:172.21.15.203:6789/0 conn(0x7f1f2013c170 0x7f1f00074dd0 :-1 s=OPENED pgs=8618 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.310+0000 7f1f1d7fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.203:6789/0 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.310+0000 7f1f1d7fa640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.310+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.203:6789/0 conn(0x7f1f2013c170 legacy=0x7f1f00074dd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.311+0000 7f1f1d7fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.311+0000 7f1f1d7fa640 10 monclient(hunting): picked mon.a con 0x7f1f200b0070 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.311+0000 7f1f1d7fa640 10 monclient(hunting): picked mon.b con 0x7f1f2006fa90 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.311+0000 7f1f1d7fa640 10 monclient(hunting): picked mon.c con 0x7f1f0007a3a0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.311+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f0007f4b0 con 0x7f1f200b0070 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.311+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f200a9b80 con 0x7f1f2006fa90 2024-03-25T19:31:43.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.311+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1f0007edb0 con 0x7f1f0007a3a0 2024-03-25T19:31:43.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.311+0000 7f1f1d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:43.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.322+0000 7f1f1d7fa640 10 client.4295.objecter ms_handle_connect 0x7f1f0007a3a0 2024-03-25T19:31:43.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.322+0000 7f1f1d7fa640 10 client.4295.objecter resend_mon_ops 2024-03-25T19:31:43.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.322+0000 7f1f1d7fa640 10 client.4295.objecter ms_handle_connect 0x7f1f200b0070 2024-03-25T19:31:43.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.322+0000 7f1f1d7fa640 10 client.4295.objecter resend_mon_ops 2024-03-25T19:31:43.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.331+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244108074 0 0) 0x7f1f00081740 con 0x7f1f0007a3a0 2024-03-25T19:31:43.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.331+0000 7f1f1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:31:43.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.331+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:43.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.331+0000 7f1f1d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:31:43.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.331+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1f0007edb0 con 0x7f1f0007a3a0 2024-03-25T19:31:43.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.337+0000 7f1f1d7fa640 10 client.4295.objecter ms_handle_connect 0x7f1f2006fa90 2024-03-25T19:31:43.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.337+0000 7f1f1d7fa640 10 client.4295.objecter resend_mon_ops 2024-03-25T19:31:43.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2936523139 0 0) 0x7f1f0007edb0 con 0x7f1f0007a3a0 2024-03-25T19:31:43.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.203:6789/0 conn(0x7f1f2006fa90 legacy=0x7f1f0007e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6789/0 conn(0x7f1f200b0070 legacy=0x7f1f000787b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 1 monclient: found mon.c 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f1f0007f000 con 0x7f1f0007a3a0 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f00081740 con 0x7f1f0007a3a0 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.345+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1f1005c820 con 0x7f1f0007a3a0 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.346+0000 7f1f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:43.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.346+0000 7f1f1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.346+0000 7f1f1d7fa640 10 monclient: dump: 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:43.347 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:43.352 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.353+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f100539e0 con 0x7f1f0007a3a0 2024-03-25T19:31:43.352 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.353+0000 7f1f1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:43.353 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.353+0000 7f1f1effd640 4 set_mon_vals no callback set 2024-03-25T19:31:43.415 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.415+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f1f100b18b0 con 0x7f1f0007a3a0 2024-03-25T19:31:43.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.419+0000 7f1f1d7fa640 1 -- 172.21.15.143:0/2291596167 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f1f100047e0 con 0x7f1f0007a3a0 2024-03-25T19:31:43.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.419+0000 7f1f1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:43.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.419+0000 7f1f1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:43.419 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:31:43.419 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 10 librados: watch_flush enter 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 10 librados: watch_flush exit 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 10 client.4295.objecter successfully canceled tick 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6800/1292282666 conn(0x7f1f0006de20 legacy=0x7f1f000702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 10 monclient: shutdown 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 >> v1:172.21.15.143:6790/0 conn(0x7f1f0007a3a0 legacy=0x7f1f0007c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 shutdown_connections 2024-03-25T19:31:43.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 >> 172.21.15.143:0/2291596167 conn(0x7f1f200b0ef0 msgr2=0x7f1f200615c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:43.423 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 shutdown_connections 2024-03-25T19:31:43.423 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 1 -- 172.21.15.143:0/2291596167 wait complete. 2024-03-25T19:31:43.423 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:43.422+0000 7f1f2658e640 1 librados: shutdown 2024-03-25T19:31:43.434 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:50.241 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.360+0000 7efff1e55640 10 monclient: get_monmap_and_config 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.360+0000 7efff1e55640 10 monclient: build_initial_monmap 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.360+0000 7efff1e55640 10 monclient: monmap: 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:31:50.361338+0000 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:31:50.361338+0000 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:31:50.360 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:31:50.361 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:50.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.361+0000 7efff1e55640 1 Processor -- start 2024-03-25T19:31:50.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.361+0000 7efff1e55640 1 -- start start 2024-03-25T19:31:50.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.361+0000 7efff1e55640 10 monclient: init 2024-03-25T19:31:50.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:50.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:50.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 10 monclient(hunting): picked mon.noname-a con 0x7effec0a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:50.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 10 monclient(hunting): picked mon.noname-c con 0x7effec13f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:50.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 10 monclient(hunting): picked mon.noname-b con 0x7effec1430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:50.363 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec142520 con 0x7effec0a98f0 2024-03-25T19:31:50.363 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec0b28a0 con 0x7effec13f710 2024-03-25T19:31:50.363 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec0b2a30 con 0x7effec1430c0 2024-03-25T19:31:50.363 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:50.363 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.362+0000 7efff1e55640 10 monclient(hunting): authenticate will time out at 6126.706109s 2024-03-25T19:31:50.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.368+0000 7effeb7fe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7effec0a98f0 0x7effec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:41750/0 (socket says 172.21.15.143:41750) 2024-03-25T19:31:50.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.368+0000 7effeb7fe640 1 -- 172.21.15.143:0/3713883604 learned_addr learned my addr 172.21.15.143:0/3713883604 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:50.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.388+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564630902 0 0) 0x7effec142520 con 0x7effec0a98f0 2024-03-25T19:31:50.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.388+0000 7effea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.388+0000 7effea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.388+0000 7effea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.388+0000 7effea7fc640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:31:50.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.388+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd4003520 con 0x7effec0a98f0 2024-03-25T19:31:50.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.397+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653763235 0 0) 0x7effec0b2a30 con 0x7effec1430c0 2024-03-25T19:31:50.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.397+0000 7effea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.397+0000 7effea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.397+0000 7effea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.397+0000 7effea7fc640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:31:50.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.397+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effec142520 con 0x7effec1430c0 2024-03-25T19:31:50.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.401+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270652071 0 0) 0x7effec0b28a0 con 0x7effec13f710 2024-03-25T19:31:50.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.401+0000 7effea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.401+0000 7effea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.401+0000 7effea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.401+0000 7effea7fc640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:31:50.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.401+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effec0b2a30 con 0x7effec13f710 2024-03-25T19:31:50.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1599415802 0 0) 0x7effd4003520 con 0x7effec0a98f0 2024-03-25T19:31:50.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 >> v1:172.21.15.143:6790/0 conn(0x7effec1430c0 legacy=0x7effec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 >> v1:172.21.15.203:6789/0 conn(0x7effec13f710 legacy=0x7effec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:50.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 1 monclient: found mon.noname-a 2024-03-25T19:31:50.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec0b44c0 con 0x7effec0a98f0 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7effdc001690 con 0x7effec0a98f0 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7effea7fc640 10 monclient: dump: 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.405 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.406 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.406 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:50.406 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:50.406 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:50.406 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:50.406 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:50.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7efff1e55640 5 monclient: authenticate success, global_id 4305 2024-03-25T19:31:50.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.404+0000 7efff1e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:50.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effdc0017f0 con 0x7effec0a98f0 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7effea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7effea7fc640 1 -- 172.21.15.143:0/3713883604 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7effdc001c00 con 0x7effec0a98f0 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7effea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7effea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7effea7fc640 10 monclient: dump: 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:50.437 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:50.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7efff1e55640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:50.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7efff1e55640 4 set_mon_vals no callback set 2024-03-25T19:31:50.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7efff1e55640 10 monclient: shutdown 2024-03-25T19:31:50.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7efff1e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:50.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.437+0000 7efff1e55640 1 -- 172.21.15.143:0/3713883604 >> v1:172.21.15.143:6789/0 conn(0x7effec0a98f0 legacy=0x7effec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.438+0000 7efff1e55640 1 -- 172.21.15.143:0/3713883604 shutdown_connections 2024-03-25T19:31:50.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.438+0000 7efff1e55640 1 -- 172.21.15.143:0/3713883604 >> 172.21.15.143:0/3713883604 conn(0x7effec0b7ea0 msgr2=0x7effec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:50.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.438+0000 7efff1e55640 1 -- 172.21.15.143:0/3713883604 shutdown_connections 2024-03-25T19:31:50.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.438+0000 7efff1e55640 1 -- 172.21.15.143:0/3713883604 wait complete. 2024-03-25T19:31:50.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 10 monclient: build_initial_monmap 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 10 monclient: monmap: 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 1 librados: starting msgr at 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 1 librados: starting objecter 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 1 Processor -- start 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 1 -- start start 2024-03-25T19:31:50.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 1 librados: setting wanted keys 2024-03-25T19:31:50.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 1 librados: calling monclient init 2024-03-25T19:31:50.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.439+0000 7efff1e55640 10 monclient: init 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 10 monclient(hunting): picked mon.noname-c con 0x7effec0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 10 monclient(hunting): picked mon.noname-a con 0x7effec13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 10 monclient(hunting): picked mon.noname-b con 0x7effec1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec132a30 con 0x7effec13f710 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec138c80 con 0x7effec1430c0 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec138f60 con 0x7effec0a98f0 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:50.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.440+0000 7efff1e55640 10 monclient(hunting): authenticate will time out at 6126.784165s 2024-03-25T19:31:50.454 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.454+0000 7effebfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7effec1430c0 0x7effec132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:34042/0 (socket says 172.21.15.143:34042) 2024-03-25T19:31:50.454 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.454+0000 7effebfff640 1 -- 172.21.15.143:0/2422377294 learned_addr learned my addr 172.21.15.143:0/2422377294 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:50.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.458+0000 7effd3fff640 10 client.?.objecter ms_handle_connect 0x7effec0a98f0 2024-03-25T19:31:50.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.458+0000 7effd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:50.460 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.461+0000 7effd3fff640 10 client.?.objecter ms_handle_connect 0x7effec1430c0 2024-03-25T19:31:50.460 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.461+0000 7effd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:50.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.468+0000 7effd3fff640 10 client.?.objecter ms_handle_connect 0x7effec13f710 2024-03-25T19:31:50.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.468+0000 7effd3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:50.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.470+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545096962 0 0) 0x7effec138f60 con 0x7effec0a98f0 2024-03-25T19:31:50.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.470+0000 7effd3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.470+0000 7effd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.470+0000 7effd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.470+0000 7effd3fff640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:31:50.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.470+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd8004eb0 con 0x7effec0a98f0 2024-03-25T19:31:50.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.474+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080674302 0 0) 0x7effec138c80 con 0x7effec1430c0 2024-03-25T19:31:50.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.474+0000 7effd3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.474+0000 7effd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.474+0000 7effd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.474+0000 7effd3fff640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:31:50.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.474+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effec138f60 con 0x7effec1430c0 2024-03-25T19:31:50.483 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.483+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159317666 0 0) 0x7effec132a30 con 0x7effec13f710 2024-03-25T19:31:50.483 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.483+0000 7effd3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:50.483 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.483+0000 7effd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:50.483 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.483+0000 7effd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:50.483 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.483+0000 7effd3fff640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:31:50.483 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.483+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effec138c80 con 0x7effec13f710 2024-03-25T19:31:50.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.489+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3841904452 0 0) 0x7effec138f60 con 0x7effec1430c0 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.489+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effec132a30 con 0x7effec1430c0 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.489+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7effe4001c10 con 0x7effec1430c0 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.489+0000 7effd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.489+0000 7effd3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.489+0000 7effd3fff640 10 monclient(hunting): dump: 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:50.489 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:50.490 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:50.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.489+0000 7effd3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.499+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 312211859 0 0) 0x7effd8004eb0 con 0x7effec0a98f0 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.499+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effec138f60 con 0x7effec0a98f0 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.499+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7effdc001b00 con 0x7effec0a98f0 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.499+0000 7effd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.499+0000 7effd3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.499+0000 7effd3fff640 10 monclient(hunting): dump: 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.499 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.500 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:50.500 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:50.500 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:50.500 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:50.500 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 642030518 0 0) 0x7effec132a30 con 0x7effec1430c0 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 >> v1:172.21.15.143:6790/0 conn(0x7effec0a98f0 legacy=0x7effec137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 >> v1:172.21.15.143:6789/0 conn(0x7effec13f710 legacy=0x7effec131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 1 monclient: found mon.b 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec139240 con 0x7effec1430c0 2024-03-25T19:31:50.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7effd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 5 monclient: authenticate success, global_id 4312 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 10 monclient: _renew_subs 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec136e50 con 0x7effec1430c0 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 10 monclient: _renew_subs 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effec1389f0 con 0x7effec1430c0 2024-03-25T19:31:50.502 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.501+0000 7efff1e55640 1 librados: init done 2024-03-25T19:31:50.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.502+0000 7efff1e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:50.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.502+0000 7efff1e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:50.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.502+0000 7efff1e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:50.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.502+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec0627b0 con 0x7effec1430c0 2024-03-25T19:31:50.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.731+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effe4003970 con 0x7effec1430c0 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.731+0000 7effd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.731+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7effe4002100 con 0x7effec1430c0 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.731+0000 7effd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.731+0000 7effd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.731+0000 7effd3fff640 10 monclient: dump: 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:50.731 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:50.732 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:50.732 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:50.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.731+0000 7effe9ffb640 4 set_mon_vals no callback set 2024-03-25T19:31:50.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.787+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7effe4003d80 con 0x7effec1430c0 2024-03-25T19:31:50.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.788+0000 7effeb7fe640 10 monclient: get_auth_request con 0x7effd806df10 auth_method 0 2024-03-25T19:31:50.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.788+0000 7effd3fff640 10 client.4312.objecter ms_handle_connect 0x7effd806df10 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.791+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7effe40549d0 con 0x7effec1430c0 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.791+0000 7effd3fff640 10 client.4312.objecter ms_dispatch 0x7effec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.791+0000 7effd3fff640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.791+0000 7effd3fff640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:50.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.791+0000 7effd3fff640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T19:31:50.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.794+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effe40504e0 con 0x7effec1430c0 2024-03-25T19:31:50.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.794+0000 7effd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:50.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:50.794+0000 7effd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:51.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.013+0000 7efff1e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.013+0000 7efff1e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.013+0000 7efff1e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:51.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.013+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effec13dd60 con 0x7effec1430c0 2024-03-25T19:31:51.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.053+0000 7effd3fff640 1 -- 172.21.15.143:0/2422377294 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7effe4052410 con 0x7effec1430c0 2024-03-25T19:31:51.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.053+0000 7effd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:51.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.053+0000 7effd3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:51.053 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:31:51.053 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:51.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 10 librados: watch_flush enter 2024-03-25T19:31:51.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 10 librados: watch_flush exit 2024-03-25T19:31:51.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T19:31:51.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 10 client.4312.objecter successfully canceled tick 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 >> v1:172.21.15.143:6800/1292282666 conn(0x7effd806df10 legacy=0x7effd80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 10 monclient: shutdown 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 >> v1:172.21.15.203:6789/0 conn(0x7effec1430c0 legacy=0x7effec132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 shutdown_connections 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 >> 172.21.15.143:0/2422377294 conn(0x7effec0b7ea0 msgr2=0x7effec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 shutdown_connections 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 1 -- 172.21.15.143:0/2422377294 wait complete. 2024-03-25T19:31:51.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:51.055+0000 7efff1e55640 1 librados: shutdown 2024-03-25T19:31:51.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:53.095 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:19.660 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:53.107 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:19.672 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:53.150 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:19.715 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:53.154 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:19.690 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:53.161 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:19.697 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:53.163 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:19.728 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:53.174 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:19.739 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:53.191 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:19.726 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:53.204 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:19.769 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:57.874 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:57.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.991+0000 7f5e11f82640 10 monclient: get_monmap_and_config 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.991+0000 7f5e11f82640 10 monclient: build_initial_monmap 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.991+0000 7f5e11f82640 10 monclient: monmap: 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:31:57.992083+0000 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:31:57.992083+0000 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:31:57.991 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:57.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 1 Processor -- start 2024-03-25T19:31:57.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 1 -- start start 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 10 monclient: init 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 10 monclient(hunting): picked mon.noname-b con 0x7f5e0c0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 10 monclient(hunting): picked mon.noname-a con 0x7f5e0c13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c142520 con 0x7f5e0c13f710 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0b28a0 con 0x7f5e0c1430c0 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0b2a30 con 0x7f5e0c0a98f0 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:57.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.992+0000 7f5e11f82640 10 monclient(hunting): authenticate will time out at 6134.336535s 2024-03-25T19:31:57.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.999+0000 7f5e0b7fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f5e0c0a98f0 0x7f5e0c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:36962/0 (socket says 172.21.15.143:36962) 2024-03-25T19:31:57.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:57.999+0000 7f5e0b7fe640 1 -- 172.21.15.143:0/1069063863 learned_addr learned my addr 172.21.15.143:0/1069063863 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:58.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.014+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214062187 0 0) 0x7f5e0c142520 con 0x7f5e0c13f710 2024-03-25T19:31:58.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.014+0000 7f5e0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.014+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.014+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.014+0000 7f5e0a7fc640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:31:58.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.014+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df4003520 con 0x7f5e0c13f710 2024-03-25T19:31:58.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.022+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603078360 0 0) 0x7f5e0c0b2a30 con 0x7f5e0c0a98f0 2024-03-25T19:31:58.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.022+0000 7f5e0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.022+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.022+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.022+0000 7f5e0a7fc640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:31:58.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.022+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c142520 con 0x7f5e0c0a98f0 2024-03-25T19:31:58.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.023+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268820832 0 0) 0x7f5e0c0b28a0 con 0x7f5e0c1430c0 2024-03-25T19:31:58.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.023+0000 7f5e0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.023+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.023+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.023+0000 7f5e0a7fc640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:31:58.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.023+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c0b2a30 con 0x7f5e0c1430c0 2024-03-25T19:31:58.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.035+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1263094969 0 0) 0x7f5e0c0b2a30 con 0x7f5e0c1430c0 2024-03-25T19:31:58.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 >> v1:172.21.15.143:6790/0 conn(0x7f5e0c0a98f0 legacy=0x7f5e0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 >> v1:172.21.15.143:6789/0 conn(0x7f5e0c13f710 legacy=0x7f5e0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c0b44c0 con 0x7f5e0c1430c0 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5e04001930 con 0x7f5e0c1430c0 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e11f82640 5 monclient: authenticate success, global_id 4315 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e11f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:58.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e0a7fc640 10 monclient: dump: 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:58.037 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.036+0000 7f5e11f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:58.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e04001a90 con 0x7f5e0c1430c0 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e0a7fc640 1 -- 172.21.15.143:0/1069063863 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5e04001ea0 con 0x7f5e0c1430c0 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e11f82640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e11f82640 4 set_mon_vals no callback set 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e0a7fc640 10 monclient: dump: 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:58.071 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:58.072 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:58.072 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:58.072 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:58.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.071+0000 7f5e11f82640 10 monclient: shutdown 2024-03-25T19:31:58.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.072+0000 7f5e11f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:58.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.072+0000 7f5e11f82640 1 -- 172.21.15.143:0/1069063863 >> v1:172.21.15.203:6789/0 conn(0x7f5e0c1430c0 legacy=0x7f5e0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.072+0000 7f5e11f82640 1 -- 172.21.15.143:0/1069063863 shutdown_connections 2024-03-25T19:31:58.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.072+0000 7f5e11f82640 1 -- 172.21.15.143:0/1069063863 >> 172.21.15.143:0/1069063863 conn(0x7f5e0c0b7ea0 msgr2=0x7f5e0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:58.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.072+0000 7f5e11f82640 1 -- 172.21.15.143:0/1069063863 shutdown_connections 2024-03-25T19:31:58.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.072+0000 7f5e11f82640 1 -- 172.21.15.143:0/1069063863 wait complete. 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 10 monclient: build_initial_monmap 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 10 monclient: monmap: 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 1 librados: starting msgr at 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 1 librados: starting objecter 2024-03-25T19:31:58.074 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 1 Processor -- start 2024-03-25T19:31:58.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 1 -- start start 2024-03-25T19:31:58.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 1 librados: setting wanted keys 2024-03-25T19:31:58.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 1 librados: calling monclient init 2024-03-25T19:31:58.075 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.073+0000 7f5e11f82640 10 monclient: init 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 10 monclient(hunting): picked mon.noname-a con 0x7f5e0c0a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 10 monclient(hunting): picked mon.noname-b con 0x7f5e0c13f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0c1430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0b1720 con 0x7f5e0c0a98f0 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c14eaf0 con 0x7f5e0c13f710 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c14ec80 con 0x7f5e0c1430c0 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:58.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.074+0000 7f5e11f82640 10 monclient(hunting): authenticate will time out at 6134.418243s 2024-03-25T19:31:58.077 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.077+0000 7f5e0b7fe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5e0c0a98f0 0x7f5e0c14d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:60962/0 (socket says 172.21.15.143:60962) 2024-03-25T19:31:58.077 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.077+0000 7f5e0b7fe640 1 -- 172.21.15.143:0/2443774593 learned_addr learned my addr 172.21.15.143:0/2443774593 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:31:58.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.083+0000 7f5df3fff640 10 client.?.objecter ms_handle_connect 0x7f5e0c13f710 2024-03-25T19:31:58.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.083+0000 7f5df3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.085+0000 7f5df3fff640 10 client.?.objecter ms_handle_connect 0x7f5e0c0a98f0 2024-03-25T19:31:58.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.085+0000 7f5df3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.086+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110452877 0 0) 0x7f5e0c14eaf0 con 0x7f5e0c13f710 2024-03-25T19:31:58.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.086+0000 7f5df3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.086+0000 7f5df3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.087+0000 7f5df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.087+0000 7f5df3fff640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:31:58.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.087+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df8004eb0 con 0x7f5e0c13f710 2024-03-25T19:31:58.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.092+0000 7f5df3fff640 10 client.?.objecter ms_handle_connect 0x7f5e0c1430c0 2024-03-25T19:31:58.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.092+0000 7f5df3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:58.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.097+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849371953 0 0) 0x7f5e0c0b1720 con 0x7f5e0c0a98f0 2024-03-25T19:31:58.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.097+0000 7f5df3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.097+0000 7f5df3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.097+0000 7f5df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.097+0000 7f5df3fff640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:31:58.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.097+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c14eaf0 con 0x7f5e0c0a98f0 2024-03-25T19:31:58.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.102+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2997361946 0 0) 0x7f5df8004eb0 con 0x7f5e0c13f710 2024-03-25T19:31:58.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.102+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e0c0b1720 con 0x7f5e0c13f710 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.102+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5e00001310 con 0x7f5e0c13f710 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.102+0000 7f5df3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.102+0000 7f5df3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.102+0000 7f5df3fff640 10 monclient(hunting): dump: 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:58.102 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:58.103 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:58.103 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.102+0000 7f5df3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:58.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.112+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253707728 0 0) 0x7f5e0c14ec80 con 0x7f5e0c1430c0 2024-03-25T19:31:58.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.112+0000 7f5df3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:58.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.112+0000 7f5df3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:58.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.112+0000 7f5df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:58.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.112+0000 7f5df3fff640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:31:58.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.112+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df8004eb0 con 0x7f5e0c1430c0 2024-03-25T19:31:58.118 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.119+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2920627108 0 0) 0x7f5e0c14eaf0 con 0x7f5e0c0a98f0 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.119+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e0c14ec80 con 0x7f5e0c0a98f0 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.119+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5dfc0019f0 con 0x7f5e0c0a98f0 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.119+0000 7f5df3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.119+0000 7f5df3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.119+0000 7f5df3fff640 10 monclient(hunting): dump: 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:58.119 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:58.120 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:58.120 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3536263111 0 0) 0x7f5e0c0b1720 con 0x7f5e0c13f710 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 >> v1:172.21.15.143:6790/0 conn(0x7f5e0c1430c0 legacy=0x7f5e0c1d4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 >> v1:172.21.15.143:6789/0 conn(0x7f5e0c0a98f0 legacy=0x7f5e0c14d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 1 monclient: found mon.b 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c14e060 con 0x7f5e0c13f710 2024-03-25T19:31:58.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5df3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 5 monclient: authenticate success, global_id 4318 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 10 monclient: _renew_subs 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c14e2d0 con 0x7f5e0c13f710 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 10 monclient: _renew_subs 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c1db460 con 0x7f5e0c13f710 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.120+0000 7f5e11f82640 1 librados: init done 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.121+0000 7f5e11f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.121+0000 7f5e11f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.121+0000 7f5e11f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:58.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.121+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c1db710 con 0x7f5e0c13f710 2024-03-25T19:31:58.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.398+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e000035f0 con 0x7f5e0c13f710 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.398+0000 7f5df3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.398+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5e00003900 con 0x7f5e0c13f710 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.398+0000 7f5df3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.399+0000 7f5df3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.399+0000 7f5df3fff640 10 monclient: dump: 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:31:58.399 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:31:58.400 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:31:58.400 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:31:58.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.399+0000 7f5e09ffb640 4 set_mon_vals no callback set 2024-03-25T19:31:58.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.400+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f5e000198d0 con 0x7f5e0c13f710 2024-03-25T19:31:58.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.400+0000 7f5e0b7fe640 10 monclient: get_auth_request con 0x7f5df806df70 auth_method 0 2024-03-25T19:31:58.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.400+0000 7f5df3fff640 10 client.4318.objecter ms_handle_connect 0x7f5df806df70 2024-03-25T19:31:58.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.404+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f5e000535e0 con 0x7f5e0c13f710 2024-03-25T19:31:58.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.404+0000 7f5df3fff640 10 client.4318.objecter ms_dispatch 0x7f5e0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:31:58.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.404+0000 7f5df3fff640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:31:58.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.404+0000 7f5df3fff640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:31:58.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.404+0000 7f5df3fff640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T19:31:58.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.405+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e0004ef20 con 0x7f5e0c13f710 2024-03-25T19:31:58.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.405+0000 7f5df3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:58.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.405+0000 7f5df3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:58.645 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.645+0000 7f5e11f82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:58.645 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.645+0000 7f5e11f82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:58.645 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.645+0000 7f5e11f82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:31:58.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.645+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e0c0627b0 con 0x7f5e0c13f710 2024-03-25T19:31:58.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.702+0000 7f5df3fff640 1 -- 172.21.15.143:0/2443774593 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f5e00051020 con 0x7f5e0c13f710 2024-03-25T19:31:58.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.702+0000 7f5df3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:58.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.702+0000 7f5df3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:58.702 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:31:58.703 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:58.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.704+0000 7f5e11f82640 10 librados: watch_flush enter 2024-03-25T19:31:58.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.704+0000 7f5e11f82640 10 librados: watch_flush exit 2024-03-25T19:31:58.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T19:31:58.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 10 client.4318.objecter successfully canceled tick 2024-03-25T19:31:58.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 >> v1:172.21.15.143:6800/1292282666 conn(0x7f5df806df70 legacy=0x7f5df8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 10 monclient: shutdown 2024-03-25T19:31:58.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:58.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 >> v1:172.21.15.203:6789/0 conn(0x7f5e0c13f710 legacy=0x7f5e0c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:58.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 shutdown_connections 2024-03-25T19:31:58.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 >> 172.21.15.143:0/2443774593 conn(0x7f5e0c0b7ea0 msgr2=0x7f5e0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:58.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 shutdown_connections 2024-03-25T19:31:58.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 1 -- 172.21.15.143:0/2443774593 wait complete. 2024-03-25T19:31:58.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:31:58.705+0000 7f5e11f82640 1 librados: shutdown 2024-03-25T19:31:58.717 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:05.523 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:05.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.621+0000 7f42e5996640 10 monclient: get_monmap_and_config 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.621+0000 7f42e5996640 10 monclient: build_initial_monmap 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.621+0000 7f42e5996640 10 monclient: monmap: 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:32:05.622263+0000 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:32:05.622263+0000 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:32:05.621 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.621+0000 7f42e5996640 1 Processor -- start 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 1 -- start start 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 10 monclient: init 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 10 monclient(hunting): picked mon.noname-b con 0x7f42e00a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 10 monclient(hunting): picked mon.noname-a con 0x7f42e013f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 10 monclient(hunting): picked mon.noname-c con 0x7f42e01430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e0142520 con 0x7f42e013f710 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00b28a0 con 0x7f42e01430c0 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00b2a30 con 0x7f42e00a98f0 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:05.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.622+0000 7f42e5996640 10 monclient(hunting): authenticate will time out at 6141.966312s 2024-03-25T19:32:05.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.624+0000 7f42deffd640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f42e00a98f0 0x7f42e00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:41772/0 (socket says 172.21.15.143:41772) 2024-03-25T19:32:05.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.624+0000 7f42deffd640 1 -- 172.21.15.143:0/2662735813 learned_addr learned my addr 172.21.15.143:0/2662735813 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:05.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.643+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081563804 0 0) 0x7f42e0142520 con 0x7f42e013f710 2024-03-25T19:32:05.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.643+0000 7f42ddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:05.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.643+0000 7f42ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:05.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.643+0000 7f42ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:05.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.643+0000 7f42ddffb640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:32:05.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.643+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c8003520 con 0x7f42e013f710 2024-03-25T19:32:05.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1248170914 0 0) 0x7f42c8003520 con 0x7f42e013f710 2024-03-25T19:32:05.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 >> v1:172.21.15.143:6790/0 conn(0x7f42e00a98f0 legacy=0x7f42e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:32:05.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 >> v1:172.21.15.203:6789/0 conn(0x7f42e01430c0 legacy=0x7f42e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:32:05.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:05.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 1 monclient: found mon.noname-a 2024-03-25T19:32:05.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:05.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e00b44c0 con 0x7f42e013f710 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f42d4001690 con 0x7f42e013f710 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42e5996640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42e5996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42ddffb640 10 monclient: dump: 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.658 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.659 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:05.659 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:05.659 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:05.659 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:05.659 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:05.659 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:05.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.657+0000 7f42e5996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:05.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.664+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42d4001860 con 0x7f42e013f710 2024-03-25T19:32:05.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.664+0000 7f42ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:05.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.664+0000 7f42ddffb640 1 -- 172.21.15.143:0/2662735813 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f42d4001cb0 con 0x7f42e013f710 2024-03-25T19:32:05.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.664+0000 7f42ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.664+0000 7f42ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.664+0000 7f42ddffb640 10 monclient: dump: 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.664+0000 7f42e5996640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:05.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.664+0000 7f42e5996640 4 set_mon_vals no callback set 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.665+0000 7f42e5996640 10 monclient: shutdown 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.665+0000 7f42e5996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.665+0000 7f42e5996640 1 -- 172.21.15.143:0/2662735813 >> v1:172.21.15.143:6789/0 conn(0x7f42e013f710 legacy=0x7f42e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.665+0000 7f42e5996640 1 -- 172.21.15.143:0/2662735813 shutdown_connections 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.665+0000 7f42e5996640 1 -- 172.21.15.143:0/2662735813 >> 172.21.15.143:0/2662735813 conn(0x7f42e00b7ea0 msgr2=0x7f42e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.665+0000 7f42e5996640 1 -- 172.21.15.143:0/2662735813 shutdown_connections 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.665+0000 7f42e5996640 1 -- 172.21.15.143:0/2662735813 wait complete. 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.666+0000 7f42e5996640 10 monclient: build_initial_monmap 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.666+0000 7f42e5996640 10 monclient: monmap: 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:05.667 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.667+0000 7f42e5996640 1 librados: starting msgr at 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.667+0000 7f42e5996640 1 librados: starting objecter 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.667+0000 7f42e5996640 1 Processor -- start 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.667+0000 7f42e5996640 1 -- start start 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.667+0000 7f42e5996640 1 librados: setting wanted keys 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.667+0000 7f42e5996640 1 librados: calling monclient init 2024-03-25T19:32:05.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.667+0000 7f42e5996640 10 monclient: init 2024-03-25T19:32:05.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:05.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:05.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 10 monclient(hunting): picked mon.noname-c con 0x7f42e00a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:05.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 10 monclient(hunting): picked mon.noname-a con 0x7f42e013f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:05.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 10 monclient(hunting): picked mon.noname-b con 0x7f42e01430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:05.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e00abe10 con 0x7f42e013f710 2024-03-25T19:32:05.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e0138df0 con 0x7f42e01430c0 2024-03-25T19:32:05.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e01d51f0 con 0x7f42e00a98f0 2024-03-25T19:32:05.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:05.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.668+0000 7f42e5996640 10 monclient(hunting): authenticate will time out at 6142.011820s 2024-03-25T19:32:05.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.681+0000 7f42de7fc640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f42e013f710 0x7f42e01d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:37344/0 (socket says 172.21.15.143:37344) 2024-03-25T19:32:05.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.681+0000 7f42de7fc640 1 -- 172.21.15.143:0/3511785978 learned_addr learned my addr 172.21.15.143:0/3511785978 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:05.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.682+0000 7f42c7fff640 10 client.?.objecter ms_handle_connect 0x7f42e013f710 2024-03-25T19:32:05.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.682+0000 7f42c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:05.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.687+0000 7f42c7fff640 10 client.?.objecter ms_handle_connect 0x7f42e01430c0 2024-03-25T19:32:05.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.687+0000 7f42c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:05.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.693+0000 7f42c7fff640 10 client.?.objecter ms_handle_connect 0x7f42e00a98f0 2024-03-25T19:32:05.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.693+0000 7f42c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:05.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.696+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122957229 0 0) 0x7f42e0138df0 con 0x7f42e01430c0 2024-03-25T19:32:05.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.697+0000 7f42c7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:05.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.697+0000 7f42c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:05.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.697+0000 7f42c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:05.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.697+0000 7f42c7fff640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:32:05.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.697+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42cc004eb0 con 0x7f42e01430c0 2024-03-25T19:32:05.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.698+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554335822 0 0) 0x7f42e00abe10 con 0x7f42e013f710 2024-03-25T19:32:05.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.698+0000 7f42c7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:05.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.698+0000 7f42c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:05.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.698+0000 7f42c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:05.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.698+0000 7f42c7fff640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:32:05.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.698+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e0138df0 con 0x7f42e013f710 2024-03-25T19:32:05.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.705+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939561733 0 0) 0x7f42e01d51f0 con 0x7f42e00a98f0 2024-03-25T19:32:05.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.705+0000 7f42c7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:05.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.705+0000 7f42c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:05.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.705+0000 7f42c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:05.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.705+0000 7f42c7fff640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:32:05.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.705+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e00abe10 con 0x7f42e00a98f0 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.718+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3327526183 0 0) 0x7f42e0138df0 con 0x7f42e013f710 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.718+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42e01d51f0 con 0x7f42e013f710 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.718+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f42d4001bb0 con 0x7f42e013f710 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.718+0000 7f42c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.718+0000 7f42c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.718+0000 7f42c7fff640 10 monclient(hunting): dump: 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:05.718 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:05.719 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:05.719 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:05.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.718+0000 7f42c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:05.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.725+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 791404922 0 0) 0x7f42e00abe10 con 0x7f42e00a98f0 2024-03-25T19:32:05.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.725+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42e0138df0 con 0x7f42e00a98f0 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.725+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f42d00014d0 con 0x7f42e00a98f0 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.725+0000 7f42c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.725+0000 7f42c7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.725+0000 7f42c7fff640 10 monclient(hunting): dump: 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:05.725 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:05.729 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.730+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3054852606 0 0) 0x7f42cc004eb0 con 0x7f42e01430c0 2024-03-25T19:32:05.729 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.730+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42e00abe10 con 0x7f42e01430c0 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.730+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f42d8001ad0 con 0x7f42e01430c0 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.730+0000 7f42c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.730+0000 7f42c7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.730+0000 7f42c7fff640 10 monclient(hunting): dump: 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:05.730 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:05.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4105346743 0 0) 0x7f42e01d51f0 con 0x7f42e013f710 2024-03-25T19:32:05.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 >> v1:172.21.15.143:6790/0 conn(0x7f42e00a98f0 legacy=0x7f42e01386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:05.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 >> v1:172.21.15.203:6789/0 conn(0x7f42e01430c0 legacy=0x7f42e01e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 1 monclient: found mon.a 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e01d83d0 con 0x7f42e013f710 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 10 monclient: _finish_auth 0 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 10 monclient: _renew_subs 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e01d77a0 con 0x7f42e013f710 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 10 monclient: _renew_subs 2024-03-25T19:32:05.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:05.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e01380e0 con 0x7f42e013f710 2024-03-25T19:32:05.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.732+0000 7f42e5996640 1 librados: init done 2024-03-25T19:32:05.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.734+0000 7f42e5996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:05.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.734+0000 7f42e5996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:05.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.734+0000 7f42e5996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:05.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:05.734+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e01d7590 con 0x7f42e013f710 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.045+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42d4003960 con 0x7f42e013f710 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.045+0000 7f42c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.045+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f42d4003d30 con 0x7f42e013f710 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.045+0000 7f42c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.045+0000 7f42c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.045+0000 7f42c7fff640 10 monclient: dump: 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:06.045 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:06.046 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:06.046 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:06.046 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:06.046 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:06.046 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:06.046 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:06.046 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.045+0000 7f42dd7fa640 4 set_mon_vals no callback set 2024-03-25T19:32:06.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.046+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f42d4003f50 con 0x7f42e013f710 2024-03-25T19:32:06.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.047+0000 7f42deffd640 10 monclient: get_auth_request con 0x7f42cc06e0f0 auth_method 0 2024-03-25T19:32:06.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.047+0000 7f42c7fff640 10 client.4320.objecter ms_handle_connect 0x7f42cc06e0f0 2024-03-25T19:32:06.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.052+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f42d4001d10 con 0x7f42e013f710 2024-03-25T19:32:06.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.052+0000 7f42c7fff640 10 client.4320.objecter ms_dispatch 0x7f42e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:06.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.052+0000 7f42c7fff640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:06.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.052+0000 7f42c7fff640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:06.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.052+0000 7f42c7fff640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-25T19:32:06.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.052+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42d4050380 con 0x7f42e013f710 2024-03-25T19:32:06.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.052+0000 7f42c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:06.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.052+0000 7f42c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:06.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.305+0000 7f42e5996640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:06.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.306+0000 7f42e5996640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:06.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.306+0000 7f42e5996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:06.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.306+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42e00627b0 con 0x7f42e013f710 2024-03-25T19:32:06.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.335+0000 7f42c7fff640 1 -- 172.21.15.143:0/3511785978 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f42d40522b0 con 0x7f42e013f710 2024-03-25T19:32:06.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.335+0000 7f42c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:06.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.335+0000 7f42c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:06.335 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:32:06.335 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 10 librados: watch_flush enter 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 10 librados: watch_flush exit 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 10 client.4320.objecter successfully canceled tick 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 >> v1:172.21.15.143:6800/1292282666 conn(0x7f42cc06e0f0 legacy=0x7f42cc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 10 monclient: shutdown 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 >> v1:172.21.15.143:6789/0 conn(0x7f42e013f710 legacy=0x7f42e01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 shutdown_connections 2024-03-25T19:32:06.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 >> 172.21.15.143:0/3511785978 conn(0x7f42e00b7ea0 msgr2=0x7f42e013ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:06.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 shutdown_connections 2024-03-25T19:32:06.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 1 -- 172.21.15.143:0/3511785978 wait complete. 2024-03-25T19:32:06.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:06.338+0000 7f42e5996640 1 librados: shutdown 2024-03-25T19:32:06.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:13.134 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:39.699 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.142 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:39.707 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.157 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:13.177 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:39.742 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.192 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:39.728 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.197 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:39.762 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.200 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:39.736 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.212 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:39.777 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.217 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:39.753 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.238 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:39.803 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:13.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.262+0000 7f9069d25640 10 monclient: get_monmap_and_config 2024-03-25T19:32:13.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.262+0000 7f9069d25640 10 monclient: build_initial_monmap 2024-03-25T19:32:13.262 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.262+0000 7f9069d25640 10 monclient: monmap: 2024-03-25T19:32:13.262 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:13.262 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:13.263 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:32:13.263848+0000 2024-03-25T19:32:13.263 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:32:13.263848+0000 2024-03-25T19:32:13.263 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:13.263 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:13.263 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:13.263 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:32:13.263 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:32:13.263 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:13.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.263+0000 7f9069d25640 1 Processor -- start 2024-03-25T19:32:13.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.263+0000 7f9069d25640 1 -- start start 2024-03-25T19:32:13.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.263+0000 7f9069d25640 10 monclient: init 2024-03-25T19:32:13.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:13.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:13.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 10 monclient(hunting): picked mon.noname-c con 0x7f90640a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:13.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 10 monclient(hunting): picked mon.noname-b con 0x7f906413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:13.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 10 monclient(hunting): picked mon.noname-a con 0x7f90641430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:13.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9064142520 con 0x7f90641430c0 2024-03-25T19:32:13.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90640b28a0 con 0x7f90640a98f0 2024-03-25T19:32:13.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90640b2a30 con 0x7f906413f710 2024-03-25T19:32:13.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:13.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.264+0000 7f9069d25640 10 monclient(hunting): authenticate will time out at 6149.608309s 2024-03-25T19:32:13.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.295+0000 7f9063fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f90641430c0 0x7f906414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:45996/0 (socket says 172.21.15.143:45996) 2024-03-25T19:32:13.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.295+0000 7f9063fff640 1 -- 172.21.15.143:0/719561998 learned_addr learned my addr 172.21.15.143:0/719561998 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:13.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.310+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957645385 0 0) 0x7f9064142520 con 0x7f90641430c0 2024-03-25T19:32:13.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.310+0000 7f90627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.310+0000 7f90627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.310+0000 7f90627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.310+0000 7f90627fc640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:32:13.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.310+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c003520 con 0x7f90641430c0 2024-03-25T19:32:13.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.325+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639983392 0 0) 0x7f90640b28a0 con 0x7f90640a98f0 2024-03-25T19:32:13.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.325+0000 7f90627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.325+0000 7f90627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.325+0000 7f90627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.325+0000 7f90627fc640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:32:13.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.325+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9064142520 con 0x7f90640a98f0 2024-03-25T19:32:13.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.336+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347361486 0 0) 0x7f90640b2a30 con 0x7f906413f710 2024-03-25T19:32:13.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.336+0000 7f90627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.336+0000 7f90627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.336+0000 7f90627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.336+0000 7f90627fc640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:32:13.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.336+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90640b28a0 con 0x7f906413f710 2024-03-25T19:32:13.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041873926 0 0) 0x7f904c003520 con 0x7f90641430c0 2024-03-25T19:32:13.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 >> v1:172.21.15.143:6790/0 conn(0x7f906413f710 legacy=0x7f90641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 >> v1:172.21.15.203:6789/0 conn(0x7f90640a98f0 legacy=0x7f90640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:13.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 1 monclient: found mon.noname-a 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90640b44c0 con 0x7f90641430c0 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f9069d25640 5 monclient: authenticate success, global_id 4323 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f9069d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f905c001a10 con 0x7f90641430c0 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f90627fc640 10 monclient: dump: 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:13.345 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.346 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.346 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.346 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:13.346 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:13.346 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:13.346 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:13.346 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:13.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.344+0000 7f9069d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:13.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f905c001b70 con 0x7f90641430c0 2024-03-25T19:32:13.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f90627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:13.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f90627fc640 1 -- 172.21.15.143:0/719561998 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f905c001f80 con 0x7f90641430c0 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f9069d25640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f9069d25640 4 set_mon_vals no callback set 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f90627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f90627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f90627fc640 10 monclient: dump: 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:13.359 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f9069d25640 10 monclient: shutdown 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f9069d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.359+0000 7f9069d25640 1 -- 172.21.15.143:0/719561998 >> v1:172.21.15.143:6789/0 conn(0x7f90641430c0 legacy=0x7f906414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.360+0000 7f9069d25640 1 -- 172.21.15.143:0/719561998 shutdown_connections 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.360+0000 7f9069d25640 1 -- 172.21.15.143:0/719561998 >> 172.21.15.143:0/719561998 conn(0x7f90640b7ea0 msgr2=0x7f90640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.360+0000 7f9069d25640 1 -- 172.21.15.143:0/719561998 shutdown_connections 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.360+0000 7f9069d25640 1 -- 172.21.15.143:0/719561998 wait complete. 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 10 monclient: build_initial_monmap 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 10 monclient: monmap: 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:13.361 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 1 librados: starting msgr at 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 1 librados: starting objecter 2024-03-25T19:32:13.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 1 Processor -- start 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 1 -- start start 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 1 librados: setting wanted keys 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 1 librados: calling monclient init 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.361+0000 7f9069d25640 10 monclient: init 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 10 monclient(hunting): picked mon.noname-b con 0x7f90640a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 10 monclient(hunting): picked mon.noname-c con 0x7f906413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 10 monclient(hunting): picked mon.noname-a con 0x7f90641430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9064136a50 con 0x7f90641430c0 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90640b79c0 con 0x7f90640a98f0 2024-03-25T19:32:13.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90640b7ca0 con 0x7f906413f710 2024-03-25T19:32:13.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:13.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.362+0000 7f9069d25640 10 monclient(hunting): authenticate will time out at 6149.706319s 2024-03-25T19:32:13.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.371+0000 7f9063fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f90641430c0 0x7f90641e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:46034/0 (socket says 172.21.15.143:46034) 2024-03-25T19:32:13.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.371+0000 7f9063fff640 1 -- 172.21.15.143:0/104914466 learned_addr learned my addr 172.21.15.143:0/104914466 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:13.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.374+0000 7f904bfff640 10 client.?.objecter ms_handle_connect 0x7f90641430c0 2024-03-25T19:32:13.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.374+0000 7f904bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:13.377 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.378+0000 7f904bfff640 10 client.?.objecter ms_handle_connect 0x7f90640a98f0 2024-03-25T19:32:13.377 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.378+0000 7f904bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:13.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.385+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569408376 0 0) 0x7f9064136a50 con 0x7f90641430c0 2024-03-25T19:32:13.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.385+0000 7f904bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.385+0000 7f904bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.385+0000 7f904bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.385+0000 7f904bfff640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:32:13.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.385+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9050004eb0 con 0x7f90641430c0 2024-03-25T19:32:13.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.388+0000 7f904bfff640 10 client.?.objecter ms_handle_connect 0x7f906413f710 2024-03-25T19:32:13.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.388+0000 7f904bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:13.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.398+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082501819 0 0) 0x7f90640b79c0 con 0x7f90640a98f0 2024-03-25T19:32:13.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.398+0000 7f904bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.398+0000 7f904bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.398+0000 7f904bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.398+0000 7f904bfff640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:32:13.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.398+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9064136a50 con 0x7f90640a98f0 2024-03-25T19:32:13.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.406+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 513468572 0 0) 0x7f9050004eb0 con 0x7f90641430c0 2024-03-25T19:32:13.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.406+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90640b79c0 con 0x7f90641430c0 2024-03-25T19:32:13.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.406+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f905c001ec0 con 0x7f90641430c0 2024-03-25T19:32:13.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.406+0000 7f904bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.406+0000 7f904bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.406+0000 7f904bfff640 10 monclient(hunting): dump: 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:13.407 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.406+0000 7f904bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:13.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.414+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26964360 0 0) 0x7f90640b7ca0 con 0x7f906413f710 2024-03-25T19:32:13.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.414+0000 7f904bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:13.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.414+0000 7f904bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:13.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.414+0000 7f904bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:13.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.414+0000 7f904bfff640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:32:13.414 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.414+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9050004eb0 con 0x7f906413f710 2024-03-25T19:32:13.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.421+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 917799939 0 0) 0x7f9064136a50 con 0x7f90640a98f0 2024-03-25T19:32:13.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.421+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90640b7ca0 con 0x7f90640a98f0 2024-03-25T19:32:13.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.421+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9054001970 con 0x7f90640a98f0 2024-03-25T19:32:13.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.421+0000 7f904bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.421+0000 7f904bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:13.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.421+0000 7f904bfff640 10 monclient(hunting): dump: 2024-03-25T19:32:13.421 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:13.421 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:13.422 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.422 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.422 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.422 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:13.422 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:13.422 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:13.422 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:13.422 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:13.423 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3378203287 0 0) 0x7f90640b79c0 con 0x7f90641430c0 2024-03-25T19:32:13.423 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 >> v1:172.21.15.143:6790/0 conn(0x7f906413f710 legacy=0x7f9064135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 >> v1:172.21.15.203:6789/0 conn(0x7f90640a98f0 legacy=0x7f9064137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 1 monclient: found mon.a 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9064136ca0 con 0x7f90641430c0 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 10 monclient: _finish_auth 0 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f904bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 5 monclient: authenticate success, global_id 4326 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 10 monclient: _renew_subs 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9064136f80 con 0x7f90641430c0 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:13.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 10 monclient: _renew_subs 2024-03-25T19:32:13.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:13.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90641360e0 con 0x7f90641430c0 2024-03-25T19:32:13.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.423+0000 7f9069d25640 1 librados: init done 2024-03-25T19:32:13.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.424+0000 7f9069d25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:13.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.424+0000 7f9069d25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:13.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.424+0000 7f9069d25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:13.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.424+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90640627b0 con 0x7f90641430c0 2024-03-25T19:32:13.749 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.749+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f905c004e10 con 0x7f90641430c0 2024-03-25T19:32:13.749 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.749+0000 7f904bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:13.749 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.749+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f905c0015d0 con 0x7f90641430c0 2024-03-25T19:32:13.749 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.749+0000 7f904bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:13.749 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.749+0000 7f904bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.749+0000 7f904bfff640 10 monclient: dump: 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:13.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.749+0000 7f9061ffb640 4 set_mon_vals no callback set 2024-03-25T19:32:13.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.750+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f905c01af90 con 0x7f90641430c0 2024-03-25T19:32:13.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.751+0000 7f90637fe640 10 monclient: get_auth_request con 0x7f905006df20 auth_method 0 2024-03-25T19:32:13.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.751+0000 7f904bfff640 10 client.4326.objecter ms_handle_connect 0x7f905006df20 2024-03-25T19:32:13.759 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.759+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f905c054bf0 con 0x7f90641430c0 2024-03-25T19:32:13.759 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.759+0000 7f904bfff640 10 client.4326.objecter ms_dispatch 0x7f9064000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:13.759 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.759+0000 7f904bfff640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:13.759 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.759+0000 7f904bfff640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:13.759 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.759+0000 7f904bfff640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-25T19:32:13.760 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.759+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f905c050530 con 0x7f90641430c0 2024-03-25T19:32:13.760 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.759+0000 7f904bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:13.760 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:13.759+0000 7f904bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:14.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.014+0000 7f9069d25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.015 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.014+0000 7f9069d25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.015 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.014+0000 7f9069d25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:14.015 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.014+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f906413e290 con 0x7f90641430c0 2024-03-25T19:32:14.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.052+0000 7f904bfff640 1 -- 172.21.15.143:0/104914466 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f905c052630 con 0x7f90641430c0 2024-03-25T19:32:14.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.052+0000 7f904bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:14.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.052+0000 7f904bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:14.052 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:32:14.053 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:14.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 10 librados: watch_flush enter 2024-03-25T19:32:14.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 10 librados: watch_flush exit 2024-03-25T19:32:14.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-25T19:32:14.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 10 client.4326.objecter successfully canceled tick 2024-03-25T19:32:14.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 >> v1:172.21.15.143:6800/1292282666 conn(0x7f905006df20 legacy=0x7f90500703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.055 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 10 monclient: shutdown 2024-03-25T19:32:14.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:14.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 >> v1:172.21.15.143:6789/0 conn(0x7f90641430c0 legacy=0x7f90641e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:14.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 shutdown_connections 2024-03-25T19:32:14.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 >> 172.21.15.143:0/104914466 conn(0x7f90640b7ea0 msgr2=0x7f9064074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:14.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 shutdown_connections 2024-03-25T19:32:14.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 1 -- 172.21.15.143:0/104914466 wait complete. 2024-03-25T19:32:14.056 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:14.055+0000 7f9069d25640 1 librados: shutdown 2024-03-25T19:32:14.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:20.873 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:20.986 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.986+0000 7fd9865d0640 10 monclient: get_monmap_and_config 2024-03-25T19:32:20.986 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.986+0000 7fd9865d0640 10 monclient: build_initial_monmap 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.987+0000 7fd9865d0640 10 monclient: monmap: 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:32:20.987932+0000 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:32:20.987932+0000 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:32:20.987 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 1 Processor -- start 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 1 -- start start 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 10 monclient: init 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd980149f50 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd98006c780 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 10 monclient(hunting): picked mon.noname-c con 0x7fd9800b1b10 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98014a330 con 0x7fd98006c780 2024-03-25T19:32:20.989 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.988+0000 7fd9865d0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd980070660 con 0x7fd9800b1b10 2024-03-25T19:32:20.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.989+0000 7fd9865d0640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98006d7c0 con 0x7fd980149f50 2024-03-25T19:32:20.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.989+0000 7fd9865d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:20.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:20.989+0000 7fd9865d0640 10 monclient(hunting): authenticate will time out at 6157.332656s 2024-03-25T19:32:20.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.000+0000 7fd97ffff640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fd980149f50 0x7fd9800b0710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:42088/0 (socket says 172.21.15.143:42088) 2024-03-25T19:32:20.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.000+0000 7fd97ffff640 1 -- 172.21.15.143:0/142307632 learned_addr learned my addr 172.21.15.143:0/142307632 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:21.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.011+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466245336 0 0) 0x7fd98014a330 con 0x7fd98006c780 2024-03-25T19:32:21.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.011+0000 7fd97effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.011+0000 7fd97effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.011+0000 7fd97effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.011+0000 7fd97effd640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:32:21.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.011+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd968003520 con 0x7fd98006c780 2024-03-25T19:32:21.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.017+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43268518 0 0) 0x7fd980070660 con 0x7fd9800b1b10 2024-03-25T19:32:21.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.017+0000 7fd97effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.017+0000 7fd97effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.017+0000 7fd97effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.017+0000 7fd97effd640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:32:21.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.017+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd98014a330 con 0x7fd9800b1b10 2024-03-25T19:32:21.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.021+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322425783 0 0) 0x7fd98006d7c0 con 0x7fd980149f50 2024-03-25T19:32:21.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.021+0000 7fd97effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.021+0000 7fd97effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.021+0000 7fd97effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.021+0000 7fd97effd640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:32:21.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.021+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd980070660 con 0x7fd980149f50 2024-03-25T19:32:21.033 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2648519670 0 0) 0x7fd968003520 con 0x7fd98006c780 2024-03-25T19:32:21.033 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 >> v1:172.21.15.143:6790/0 conn(0x7fd980149f50 legacy=0x7fd9800b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.033 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 >> v1:172.21.15.203:6789/0 conn(0x7fd9800b1b10 legacy=0x7fd980152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.033 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:21.033 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 1 monclient: found mon.noname-a 2024-03-25T19:32:21.033 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98006d3e0 con 0x7fd98006c780 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 10 monclient: _finish_auth 0 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd974001690 con 0x7fd98006c780 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd9865d0640 5 monclient: authenticate success, global_id 4329 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd9865d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd97effd640 10 monclient: dump: 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:21.034 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:21.035 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:21.035 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.035 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:21.035 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:21.035 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:21.035 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:21.035 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:21.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.033+0000 7fd9865d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:21.063 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.063+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9740017f0 con 0x7fd98006c780 2024-03-25T19:32:21.063 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.063+0000 7fd97effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:21.063 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.063+0000 7fd97effd640 1 -- 172.21.15.143:0/142307632 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd974001c00 con 0x7fd98006c780 2024-03-25T19:32:21.063 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.063+0000 7fd97effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.063+0000 7fd97effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.063+0000 7fd97effd640 10 monclient: dump: 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:21.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.063+0000 7fd9865d0640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:21.065 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.063+0000 7fd9865d0640 4 set_mon_vals no callback set 2024-03-25T19:32:21.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.064+0000 7fd9865d0640 10 monclient: shutdown 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.064+0000 7fd9865d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.064+0000 7fd9865d0640 1 -- 172.21.15.143:0/142307632 >> v1:172.21.15.143:6789/0 conn(0x7fd98006c780 legacy=0x7fd9800b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.064+0000 7fd9865d0640 1 -- 172.21.15.143:0/142307632 shutdown_connections 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.064+0000 7fd9865d0640 1 -- 172.21.15.143:0/142307632 >> 172.21.15.143:0/142307632 conn(0x7fd9800b5d10 msgr2=0x7fd9800b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.064+0000 7fd9865d0640 1 -- 172.21.15.143:0/142307632 shutdown_connections 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.064+0000 7fd9865d0640 1 -- 172.21.15.143:0/142307632 wait complete. 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.065+0000 7fd9865d0640 10 monclient: build_initial_monmap 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.065+0000 7fd9865d0640 10 monclient: monmap: 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:21.067 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.065+0000 7fd9865d0640 1 librados: starting msgr at 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.065+0000 7fd9865d0640 1 librados: starting objecter 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.066+0000 7fd9865d0640 1 Processor -- start 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.066+0000 7fd9865d0640 1 -- start start 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.066+0000 7fd9865d0640 1 librados: setting wanted keys 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.066+0000 7fd9865d0640 1 librados: calling monclient init 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.066+0000 7fd9865d0640 10 monclient: init 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd98006c780 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:21.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 10 monclient(hunting): picked mon.noname-c con 0x7fd9800b1b10 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:21.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd980149f50 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:21.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9800ad3c0 con 0x7fd980149f50 2024-03-25T19:32:21.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9801d9630 con 0x7fd98006c780 2024-03-25T19:32:21.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9800af960 con 0x7fd9800b1b10 2024-03-25T19:32:21.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:21.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.067+0000 7fd9865d0640 10 monclient(hunting): authenticate will time out at 6157.411034s 2024-03-25T19:32:21.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.072+0000 7fd97f7fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fd9800b1b10 0x7fd9800ad850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:42156/0 (socket says 172.21.15.143:42156) 2024-03-25T19:32:21.072 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.072+0000 7fd97f7fe640 1 -- 172.21.15.143:0/3239330433 learned_addr learned my addr 172.21.15.143:0/3239330433 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:21.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.077+0000 7fd97cff9640 10 client.?.objecter ms_handle_connect 0x7fd9800b1b10 2024-03-25T19:32:21.076 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.077+0000 7fd97cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.084 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.084+0000 7fd97cff9640 10 client.?.objecter ms_handle_connect 0x7fd98006c780 2024-03-25T19:32:21.084 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.084+0000 7fd97cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.087+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480953778 0 0) 0x7fd9800af960 con 0x7fd9800b1b10 2024-03-25T19:32:21.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.087+0000 7fd97cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.087+0000 7fd97cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.087+0000 7fd97cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.087+0000 7fd97cff9640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:32:21.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.087+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd96c004eb0 con 0x7fd9800b1b10 2024-03-25T19:32:21.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.094+0000 7fd97cff9640 10 client.?.objecter ms_handle_connect 0x7fd980149f50 2024-03-25T19:32:21.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.094+0000 7fd97cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:21.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.097+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500625748 0 0) 0x7fd96c004eb0 con 0x7fd9800b1b10 2024-03-25T19:32:21.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.097+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9800af960 con 0x7fd9800b1b10 2024-03-25T19:32:21.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.097+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd974001bc0 con 0x7fd9800b1b10 2024-03-25T19:32:21.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.097+0000 7fd97cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.097+0000 7fd97cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:21.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.097+0000 7fd97cff9640 10 monclient(hunting): dump: 2024-03-25T19:32:21.097 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:21.097 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:21.098 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.097+0000 7fd97cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:21.102 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.103+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476138121 0 0) 0x7fd9801d9630 con 0x7fd98006c780 2024-03-25T19:32:21.103 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.103+0000 7fd97cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:21.103 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.103+0000 7fd97cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:21.103 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.103+0000 7fd97cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:21.103 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.103+0000 7fd97cff9640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:32:21.103 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.103+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd96c004eb0 con 0x7fd98006c780 2024-03-25T19:32:21.104 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1032797733 0 0) 0x7fd9800af960 con 0x7fd9800b1b10 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 >> v1:172.21.15.203:6789/0 conn(0x7fd98006c780 legacy=0x7fd9801d8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 >> v1:172.21.15.143:6789/0 conn(0x7fd980149f50 legacy=0x7fd9800adf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 1 monclient: found mon.c 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9801428f0 con 0x7fd9800b1b10 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 10 monclient: _finish_auth 0 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd97cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 5 monclient: authenticate success, global_id 4322 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 10 monclient: _renew_subs 2024-03-25T19:32:21.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9801e7310 con 0x7fd9800b1b10 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 10 monclient: _renew_subs 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9801e7680 con 0x7fd9800b1b10 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.104+0000 7fd9865d0640 1 librados: init done 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.105+0000 7fd9865d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.105+0000 7fd9865d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.105+0000 7fd9865d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:21.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.105+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9800b15e0 con 0x7fd9800b1b10 2024-03-25T19:32:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.401+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9740011f0 con 0x7fd9800b1b10 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.401+0000 7fd97cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.402+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd974003b00 con 0x7fd9800b1b10 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.402+0000 7fd97cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.402+0000 7fd97cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.402+0000 7fd97cff9640 10 monclient: dump: 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:21.402 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.402+0000 7fd97e7fc640 4 set_mon_vals no callback set 2024-03-25T19:32:21.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.403+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fd974003d20 con 0x7fd9800b1b10 2024-03-25T19:32:21.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.403+0000 7fd97ffff640 10 monclient: get_auth_request con 0x7fd96c06dce0 auth_method 0 2024-03-25T19:32:21.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.403+0000 7fd97cff9640 10 client.4322.objecter ms_handle_connect 0x7fd96c06dce0 2024-03-25T19:32:21.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.408+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fd974054a00 con 0x7fd9800b1b10 2024-03-25T19:32:21.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.408+0000 7fd97cff9640 10 client.4322.objecter ms_dispatch 0x7fd980000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:21.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.408+0000 7fd97cff9640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:21.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.408+0000 7fd97cff9640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:21.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.408+0000 7fd97cff9640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-25T19:32:21.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.408+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd974050340 con 0x7fd9800b1b10 2024-03-25T19:32:21.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.408+0000 7fd97cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:21.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.408+0000 7fd97cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:21.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.655+0000 7fd9865d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:21.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.655+0000 7fd9865d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:21.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.655+0000 7fd9865d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:21.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.655+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9800627b0 con 0x7fd9800b1b10 2024-03-25T19:32:21.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.679+0000 7fd97cff9640 1 -- 172.21.15.143:0/3239330433 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fd974052440 con 0x7fd9800b1b10 2024-03-25T19:32:21.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.679+0000 7fd97cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:21.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.679+0000 7fd97cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:21.679 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:32:21.679 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 10 librados: watch_flush enter 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 10 librados: watch_flush exit 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 10 client.4322.objecter successfully canceled tick 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 >> v1:172.21.15.143:6800/1292282666 conn(0x7fd96c06dce0 legacy=0x7fd96c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 10 monclient: shutdown 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 >> v1:172.21.15.143:6790/0 conn(0x7fd9800b1b10 legacy=0x7fd9800ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:21.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 shutdown_connections 2024-03-25T19:32:21.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 >> 172.21.15.143:0/3239330433 conn(0x7fd9800b5d10 msgr2=0x7fd9800b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:21.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 shutdown_connections 2024-03-25T19:32:21.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 1 -- 172.21.15.143:0/3239330433 wait complete. 2024-03-25T19:32:21.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:21.682+0000 7fd9865d0640 1 librados: shutdown 2024-03-25T19:32:21.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:28.499 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:28.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.599+0000 7f3c17893640 10 monclient: get_monmap_and_config 2024-03-25T19:32:28.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.599+0000 7f3c17893640 10 monclient: build_initial_monmap 2024-03-25T19:32:28.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.599+0000 7f3c17893640 10 monclient: monmap: 2024-03-25T19:32:28.598 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:32:28.599976+0000 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:32:28.599976+0000 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:32:28.599 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:28.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.599+0000 7f3c17893640 1 Processor -- start 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.599+0000 7f3c17893640 1 -- start start 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.599+0000 7f3c17893640 10 monclient: init 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 10 monclient(hunting): picked mon.noname-b con 0x7f3c100b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 10 monclient(hunting): picked mon.noname-a con 0x7f3c100b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 10 monclient(hunting): picked mon.noname-c con 0x7f3c1014be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c100af940 con 0x7f3c100b1730 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c100b0a70 con 0x7f3c1014be90 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c100b0d50 con 0x7f3c100b4d60 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.600+0000 7f3c17893640 10 monclient(hunting): authenticate will time out at 6164.944147s 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.601+0000 7f3c15e30640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3c1014be90 0x7f3c1014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:42430/0 (socket says 172.21.15.143:42430) 2024-03-25T19:32:28.601 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.601+0000 7f3c15e30640 1 -- 172.21.15.143:0/1397920746 learned_addr learned my addr 172.21.15.143:0/1397920746 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:28.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.634+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606903905 0 0) 0x7f3c100b0a70 con 0x7f3c1014be90 2024-03-25T19:32:28.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.634+0000 7f3c07fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:28.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.634+0000 7f3c07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:28.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.634+0000 7f3c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:28.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.634+0000 7f3c07fff640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:32:28.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.634+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc003520 con 0x7f3c1014be90 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2407018488 0 0) 0x7f3bfc003520 con 0x7f3c1014be90 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 >> v1:172.21.15.143:6790/0 conn(0x7f3c100b4d60 legacy=0x7f3c1013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 >> v1:172.21.15.143:6789/0 conn(0x7f3c100b1730 legacy=0x7f3c100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 1 monclient: found mon.noname-c 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c100b1030 con 0x7f3c1014be90 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 10 monclient: _finish_auth 0 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c17893640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:32:28.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3c0c001670 con 0x7f3c1014be90 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c07fff640 10 monclient: dump: 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:28.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.642+0000 7f3c17893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.661+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c0c001840 con 0x7f3c1014be90 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.661+0000 7f3c07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.661+0000 7f3c07fff640 1 -- 172.21.15.143:0/1397920746 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3c0c001c50 con 0x7f3c1014be90 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.661+0000 7f3c17893640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.661+0000 7f3c17893640 4 set_mon_vals no callback set 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.661+0000 7f3c07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.661+0000 7f3c07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.661+0000 7f3c07fff640 10 monclient: dump: 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:28.661 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:28.662 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:28.662 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:28.662 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:28.662 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:28.662 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:28.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.662+0000 7f3c17893640 10 monclient: shutdown 2024-03-25T19:32:28.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.662+0000 7f3c17893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:28.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.662+0000 7f3c17893640 1 -- 172.21.15.143:0/1397920746 >> v1:172.21.15.203:6789/0 conn(0x7f3c1014be90 legacy=0x7f3c1014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:28.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.662+0000 7f3c17893640 1 -- 172.21.15.143:0/1397920746 shutdown_connections 2024-03-25T19:32:28.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.662+0000 7f3c17893640 1 -- 172.21.15.143:0/1397920746 >> 172.21.15.143:0/1397920746 conn(0x7f3c100617a0 msgr2=0x7f3c10061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.662+0000 7f3c17893640 1 -- 172.21.15.143:0/1397920746 shutdown_connections 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.662+0000 7f3c17893640 1 -- 172.21.15.143:0/1397920746 wait complete. 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 10 monclient: build_initial_monmap 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 10 monclient: monmap: 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:28.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 1 librados: starting msgr at 2024-03-25T19:32:28.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 1 librados: starting objecter 2024-03-25T19:32:28.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 1 Processor -- start 2024-03-25T19:32:28.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 1 -- start start 2024-03-25T19:32:28.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 1 librados: setting wanted keys 2024-03-25T19:32:28.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 1 librados: calling monclient init 2024-03-25T19:32:28.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.663+0000 7f3c17893640 10 monclient: init 2024-03-25T19:32:28.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:28.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:32:28.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 10 monclient(hunting): picked mon.noname-b con 0x7f3c100b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:28.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 10 monclient(hunting): picked mon.noname-a con 0x7f3c100b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:28.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 10 monclient(hunting): picked mon.noname-c con 0x7f3c1014be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:28.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c10137c70 con 0x7f3c100b4d60 2024-03-25T19:32:28.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c10138940 con 0x7f3c100b1730 2024-03-25T19:32:28.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c10140520 con 0x7f3c1014be90 2024-03-25T19:32:28.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:28.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.664+0000 7f3c17893640 10 monclient(hunting): authenticate will time out at 6165.008637s 2024-03-25T19:32:28.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.675+0000 7f3c1562f640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3c100b1730 0x7f3c10140940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:42444/0 (socket says 172.21.15.143:42444) 2024-03-25T19:32:28.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.675+0000 7f3c1562f640 1 -- 172.21.15.143:0/323881929 learned_addr learned my addr 172.21.15.143:0/323881929 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:28.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.676+0000 7f3c05ffb640 10 client.?.objecter ms_handle_connect 0x7f3c100b1730 2024-03-25T19:32:28.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.676+0000 7f3c05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:28.687 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.688+0000 7f3c05ffb640 10 client.?.objecter ms_handle_connect 0x7f3c100b4d60 2024-03-25T19:32:28.687 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.688+0000 7f3c05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:28.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.691+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137356327 0 0) 0x7f3c10138940 con 0x7f3c100b1730 2024-03-25T19:32:28.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.691+0000 7f3c05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:28.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.691+0000 7f3c05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:28.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.691+0000 7f3c05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:28.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.691+0000 7f3c05ffb640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:32:28.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.691+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf4004eb0 con 0x7f3c100b1730 2024-03-25T19:32:28.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.693+0000 7f3c05ffb640 10 client.?.objecter ms_handle_connect 0x7f3c1014be90 2024-03-25T19:32:28.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.693+0000 7f3c05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:28.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.699+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460841617 0 0) 0x7f3c10137c70 con 0x7f3c100b4d60 2024-03-25T19:32:28.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.699+0000 7f3c05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:28.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.699+0000 7f3c05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:28.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.699+0000 7f3c05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:28.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.699+0000 7f3c05ffb640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:32:28.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.699+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c10138940 con 0x7f3c100b4d60 2024-03-25T19:32:28.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.707+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 455038130 0 0) 0x7f3bf4004eb0 con 0x7f3c100b1730 2024-03-25T19:32:28.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.708+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c10137c70 con 0x7f3c100b1730 2024-03-25T19:32:28.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.708+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3c08001db0 con 0x7f3c100b1730 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.708+0000 7f3c05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.708+0000 7f3c05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.708+0000 7f3c05ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:28.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.708+0000 7f3c05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:28.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.712+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002071064 0 0) 0x7f3c10140520 con 0x7f3c1014be90 2024-03-25T19:32:28.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.712+0000 7f3c05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:28.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.712+0000 7f3c05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:28.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.712+0000 7f3c05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:28.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.712+0000 7f3c05ffb640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:32:28.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.712+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf4004eb0 con 0x7f3c1014be90 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.713+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653819804 0 0) 0x7f3c10138940 con 0x7f3c100b4d60 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.713+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c10140520 con 0x7f3c100b4d60 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.713+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3c00001810 con 0x7f3c100b4d60 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.713+0000 7f3c05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.713+0000 7f3c05ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.713+0000 7f3c05ffb640 10 monclient(hunting): dump: 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:28.713 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:28.714 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:28.714 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:28.714 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:28.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 829895293 0 0) 0x7f3c10140520 con 0x7f3c100b4d60 2024-03-25T19:32:28.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 >> v1:172.21.15.143:6790/0 conn(0x7f3c1014be90 legacy=0x7f3c101e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:28.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 >> v1:172.21.15.203:6789/0 conn(0x7f3c100b1730 legacy=0x7f3c10140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:28.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 1 monclient: found mon.a 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c10136e80 con 0x7f3c100b4d60 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c17893640 5 monclient: authenticate success, global_id 4335 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c17893640 10 monclient: _renew_subs 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c17893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c10145250 con 0x7f3c100b4d60 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.719+0000 7f3c17893640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.720+0000 7f3c17893640 10 monclient: _renew_subs 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.720+0000 7f3c17893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:28.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.720+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c10136a80 con 0x7f3c100b4d60 2024-03-25T19:32:28.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.720+0000 7f3c17893640 1 librados: init done 2024-03-25T19:32:28.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.721+0000 7f3c17893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:28.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.721+0000 7f3c17893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:28.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.721+0000 7f3c17893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:28.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:28.721+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c100627b0 con 0x7f3c100b4d60 2024-03-25T19:32:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.017+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c00001d00 con 0x7f3c100b4d60 2024-03-25T19:32:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.017+0000 7f3c05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.017+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3c00003790 con 0x7f3c100b4d60 2024-03-25T19:32:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.017+0000 7f3c05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.017+0000 7f3c05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.017+0000 7f3c05ffb640 10 monclient: dump: 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:29.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.017+0000 7f3c077fe640 4 set_mon_vals no callback set 2024-03-25T19:32:29.019 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.018+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f3c000197a0 con 0x7f3c100b4d60 2024-03-25T19:32:29.019 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.019+0000 7f3c1562f640 10 monclient: get_auth_request con 0x7f3bf406df70 auth_method 0 2024-03-25T19:32:29.019 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.019+0000 7f3c05ffb640 10 client.4335.objecter ms_handle_connect 0x7f3bf406df70 2024-03-25T19:32:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.024+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f3c00053510 con 0x7f3c100b4d60 2024-03-25T19:32:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.024+0000 7f3c05ffb640 10 client.4335.objecter ms_dispatch 0x7f3c10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.024+0000 7f3c05ffb640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.024+0000 7f3c05ffb640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.024+0000 7f3c05ffb640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-25T19:32:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.024+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c0004ee50 con 0x7f3c100b4d60 2024-03-25T19:32:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.024+0000 7f3c05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.024+0000 7f3c05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:29.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.277+0000 7f3c17893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:29.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.277+0000 7f3c17893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:29.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.277+0000 7f3c17893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:29.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.277+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c10069f40 con 0x7f3c100b4d60 2024-03-25T19:32:29.299 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.300+0000 7f3c05ffb640 1 -- 172.21.15.143:0/323881929 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f3c00050f50 con 0x7f3c100b4d60 2024-03-25T19:32:29.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.300+0000 7f3c05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:29.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.300+0000 7f3c05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:29.300 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:32:29.300 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.302+0000 7f3c17893640 10 librados: watch_flush enter 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.302+0000 7f3c17893640 10 librados: watch_flush exit 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.302+0000 7f3c17893640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.302+0000 7f3c17893640 10 client.4335.objecter successfully canceled tick 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 >> v1:172.21.15.143:6800/1292282666 conn(0x7f3bf406df70 legacy=0x7f3bf4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 10 monclient: shutdown 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 >> v1:172.21.15.143:6789/0 conn(0x7f3c100b4d60 legacy=0x7f3c1012fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 shutdown_connections 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 >> 172.21.15.143:0/323881929 conn(0x7f3c100617a0 msgr2=0x7f3c10061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 shutdown_connections 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 1 -- 172.21.15.143:0/323881929 wait complete. 2024-03-25T19:32:29.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:29.303+0000 7f3c17893640 1 librados: shutdown 2024-03-25T19:32:29.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:33.185 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:59.750 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:33.190 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:59.755 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:33.227 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:02:59.792 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:33.249 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:59.814 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:33.250 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:59.785 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:33.254 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:59.790 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:33.262 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:59.827 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:33.291 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:02:59.827 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:33.297 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:02:59.862 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:36.122 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:36.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.242+0000 7f16dea25640 10 monclient: get_monmap_and_config 2024-03-25T19:32:36.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.242+0000 7f16dea25640 10 monclient: build_initial_monmap 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.242+0000 7f16dea25640 10 monclient: monmap: 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:32:36.242993+0000 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:32:36.242993+0000 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:32:36.242 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:36.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.242+0000 7f16dea25640 1 Processor -- start 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 1 -- start start 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 10 monclient: init 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 10 monclient(hunting): picked mon.noname-c con 0x7f16d80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 10 monclient(hunting): picked mon.noname-b con 0x7f16d813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 10 monclient(hunting): picked mon.noname-a con 0x7f16d81430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d8142520 con 0x7f16d81430c0 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d80b28a0 con 0x7f16d80a98f0 2024-03-25T19:32:36.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d80b2a30 con 0x7f16d813f710 2024-03-25T19:32:36.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:36.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.243+0000 7f16dea25640 10 monclient(hunting): authenticate will time out at 6172.587301s 2024-03-25T19:32:36.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.251+0000 7f16d7fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f16d80a98f0 0x7f16d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:47166/0 (socket says 172.21.15.143:47166) 2024-03-25T19:32:36.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.251+0000 7f16d7fff640 1 -- 172.21.15.143:0/731184530 learned_addr learned my addr 172.21.15.143:0/731184530 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:36.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.261+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992558279 0 0) 0x7f16d80b28a0 con 0x7f16d80a98f0 2024-03-25T19:32:36.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.261+0000 7f16d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:36.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.261+0000 7f16d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:36.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.261+0000 7f16d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:36.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.261+0000 7f16d6ffd640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:32:36.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.261+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16c0003520 con 0x7f16d80a98f0 2024-03-25T19:32:36.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257408505 0 0) 0x7f16d8142520 con 0x7f16d81430c0 2024-03-25T19:32:36.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:36.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:36.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d80b28a0 con 0x7f16d81430c0 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1537081989 0 0) 0x7f16c0003520 con 0x7f16d80a98f0 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 >> v1:172.21.15.143:6790/0 conn(0x7f16d813f710 legacy=0x7f16d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 >> v1:172.21.15.143:6789/0 conn(0x7f16d81430c0 legacy=0x7f16d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d80b44c0 con 0x7f16d80a98f0 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:36.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f16c8001690 con 0x7f16d80a98f0 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16dea25640 5 monclient: authenticate success, global_id 4342 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16dea25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16d6ffd640 10 monclient: dump: 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:36.272 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:36.273 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:36.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.270+0000 7f16dea25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:36.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16c80017f0 con 0x7f16d80a98f0 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16d6ffd640 1 -- 172.21.15.143:0/731184530 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f16c8001c00 con 0x7f16d80a98f0 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16d6ffd640 10 monclient: dump: 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:36.291 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:36.292 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:36.292 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:36.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16dea25640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:36.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16dea25640 4 set_mon_vals no callback set 2024-03-25T19:32:36.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16dea25640 10 monclient: shutdown 2024-03-25T19:32:36.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16dea25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:36.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.291+0000 7f16dea25640 1 -- 172.21.15.143:0/731184530 >> v1:172.21.15.203:6789/0 conn(0x7f16d80a98f0 legacy=0x7f16d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.292+0000 7f16dea25640 1 -- 172.21.15.143:0/731184530 shutdown_connections 2024-03-25T19:32:36.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.292+0000 7f16dea25640 1 -- 172.21.15.143:0/731184530 >> 172.21.15.143:0/731184530 conn(0x7f16d80b7ea0 msgr2=0x7f16d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:36.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.292+0000 7f16dea25640 1 -- 172.21.15.143:0/731184530 shutdown_connections 2024-03-25T19:32:36.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.292+0000 7f16dea25640 1 -- 172.21.15.143:0/731184530 wait complete. 2024-03-25T19:32:36.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.292+0000 7f16dea25640 10 monclient: build_initial_monmap 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.292+0000 7f16dea25640 10 monclient: monmap: 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.293+0000 7f16dea25640 1 librados: starting msgr at 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.293+0000 7f16dea25640 1 librados: starting objecter 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.293+0000 7f16dea25640 1 Processor -- start 2024-03-25T19:32:36.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.293+0000 7f16dea25640 1 -- start start 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.293+0000 7f16dea25640 1 librados: setting wanted keys 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.293+0000 7f16dea25640 1 librados: calling monclient init 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.293+0000 7f16dea25640 10 monclient: init 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 10 monclient(hunting): picked mon.noname-a con 0x7f16d80a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 10 monclient(hunting): picked mon.noname-c con 0x7f16d813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 10 monclient(hunting): picked mon.noname-b con 0x7f16d81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d8138250 con 0x7f16d80a98f0 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d8137d50 con 0x7f16d81430c0 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16d8137ee0 con 0x7f16d813f710 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.294+0000 7f16dea25640 10 monclient(hunting): authenticate will time out at 6172.638068s 2024-03-25T19:32:36.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.301+0000 7f16d77fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f16d813f710 0x7f16d81d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:33206/0 (socket says 172.21.15.143:33206) 2024-03-25T19:32:36.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.301+0000 7f16d77fe640 1 -- 172.21.15.143:0/1329439735 learned_addr learned my addr 172.21.15.143:0/1329439735 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:36.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.302+0000 7f16d4ff9640 10 client.?.objecter ms_handle_connect 0x7f16d813f710 2024-03-25T19:32:36.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.302+0000 7f16d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:36.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.305+0000 7f16d4ff9640 10 client.?.objecter ms_handle_connect 0x7f16d81430c0 2024-03-25T19:32:36.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.305+0000 7f16d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:36.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.312+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774788322 0 0) 0x7f16d8137d50 con 0x7f16d81430c0 2024-03-25T19:32:36.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.312+0000 7f16d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:36.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.312+0000 7f16d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:36.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.312+0000 7f16d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:36.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.312+0000 7f16d4ff9640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:32:36.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.312+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16c4004eb0 con 0x7f16d81430c0 2024-03-25T19:32:36.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.312+0000 7f16d4ff9640 10 client.?.objecter ms_handle_connect 0x7f16d80a98f0 2024-03-25T19:32:36.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.312+0000 7f16d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:36.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.313+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821897512 0 0) 0x7f16d8137ee0 con 0x7f16d813f710 2024-03-25T19:32:36.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.313+0000 7f16d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:36.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.313+0000 7f16d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:36.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.313+0000 7f16d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:36.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.313+0000 7f16d4ff9640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:32:36.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.313+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8137d50 con 0x7f16d813f710 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.325+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 980646995 0 0) 0x7f16c4004eb0 con 0x7f16d81430c0 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.325+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16d8137ee0 con 0x7f16d81430c0 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.325+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f16d0001bd0 con 0x7f16d81430c0 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.325+0000 7f16d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.325+0000 7f16d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.325+0000 7f16d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.325 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.326 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:36.326 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:36.326 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:36.326 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:36.326 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:36.326 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:36.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.325+0000 7f16d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.327+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2185579652 0 0) 0x7f16d8137d50 con 0x7f16d813f710 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.327+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16c4004eb0 con 0x7f16d813f710 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.327+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f16cc001970 con 0x7f16d813f710 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.327+0000 7f16d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.327+0000 7f16d4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.327+0000 7f16d4ff9640 10 monclient(hunting): dump: 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.327 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.328 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:36.328 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:36.328 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:36.328 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:36.328 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:36.328 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:36.334 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2503810329 0 0) 0x7f16c4004eb0 con 0x7f16d813f710 2024-03-25T19:32:36.334 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 >> v1:172.21.15.203:6789/0 conn(0x7f16d81430c0 legacy=0x7f16d81d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.334 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 >> v1:172.21.15.143:6789/0 conn(0x7f16d80a98f0 legacy=0x7f16d8137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.334 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:32:36.334 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 1 monclient: found mon.c 2024-03-25T19:32:36.334 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:36.334 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:36.334 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d80b7730 con 0x7f16d813f710 2024-03-25T19:32:36.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:32:36.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:32:36.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:36.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16dea25640 5 monclient: authenticate success, global_id 4334 2024-03-25T19:32:36.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16dea25640 10 monclient: _renew_subs 2024-03-25T19:32:36.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16dea25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:36.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.333+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d80b7920 con 0x7f16d813f710 2024-03-25T19:32:36.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.334+0000 7f16dea25640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:36.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.334+0000 7f16dea25640 10 monclient: _renew_subs 2024-03-25T19:32:36.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.334+0000 7f16dea25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:36.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.334+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d81e33d0 con 0x7f16d813f710 2024-03-25T19:32:36.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.334+0000 7f16dea25640 1 librados: init done 2024-03-25T19:32:36.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.335+0000 7f16dea25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:36.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.335+0000 7f16dea25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:36.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.335+0000 7f16dea25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:36.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.335+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d80627b0 con 0x7f16d813f710 2024-03-25T19:32:36.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.356+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16cc003610 con 0x7f16d813f710 2024-03-25T19:32:36.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.356+0000 7f16d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:36.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.356+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f16cc0039a0 con 0x7f16d813f710 2024-03-25T19:32:36.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.356+0000 7f16d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:36.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.356+0000 7f16d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:36.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.356+0000 7f16d4ff9640 10 monclient: dump: 2024-03-25T19:32:36.356 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:36.356 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:36.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.356+0000 7f16d67fc640 4 set_mon_vals no callback set 2024-03-25T19:32:36.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.357+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f16cc003bc0 con 0x7f16d813f710 2024-03-25T19:32:36.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.357+0000 7f16d7fff640 10 monclient: get_auth_request con 0x7f16c406de80 auth_method 0 2024-03-25T19:32:36.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.357+0000 7f16d4ff9640 10 client.4334.objecter ms_handle_connect 0x7f16c406de80 2024-03-25T19:32:36.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.361+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f16cc054840 con 0x7f16d813f710 2024-03-25T19:32:36.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.361+0000 7f16d4ff9640 10 client.4334.objecter ms_dispatch 0x7f16d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:36.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.361+0000 7f16d4ff9640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:36.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.361+0000 7f16d4ff9640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:36.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.361+0000 7f16d4ff9640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-25T19:32:36.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.657+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16cc050350 con 0x7f16d813f710 2024-03-25T19:32:36.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.657+0000 7f16d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:36.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.657+0000 7f16d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:36.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.887+0000 7f16dea25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:36.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.887+0000 7f16dea25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:36.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.887+0000 7f16dea25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:36.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.887+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16d81383e0 con 0x7f16d813f710 2024-03-25T19:32:36.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.915+0000 7f16d4ff9640 1 -- 172.21.15.143:0/1329439735 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f16cc052280 con 0x7f16d813f710 2024-03-25T19:32:36.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.915+0000 7f16d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:36.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.915+0000 7f16d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:36.915 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:32:36.915 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:36.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 10 librados: watch_flush enter 2024-03-25T19:32:36.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 10 librados: watch_flush exit 2024-03-25T19:32:36.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-25T19:32:36.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 10 client.4334.objecter successfully canceled tick 2024-03-25T19:32:36.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 >> v1:172.21.15.143:6800/1292282666 conn(0x7f16c406de80 legacy=0x7f16c4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 10 monclient: shutdown 2024-03-25T19:32:36.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:36.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 >> v1:172.21.15.143:6790/0 conn(0x7f16d813f710 legacy=0x7f16d81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:36.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 shutdown_connections 2024-03-25T19:32:36.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 >> 172.21.15.143:0/1329439735 conn(0x7f16d80b7ea0 msgr2=0x7f16d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:36.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 shutdown_connections 2024-03-25T19:32:36.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 1 -- 172.21.15.143:0/1329439735 wait complete. 2024-03-25T19:32:36.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:36.917+0000 7f16dea25640 1 librados: shutdown 2024-03-25T19:32:36.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:43.734 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:43.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.851+0000 7f7fa6925640 10 monclient: get_monmap_and_config 2024-03-25T19:32:43.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.851+0000 7f7fa6925640 10 monclient: build_initial_monmap 2024-03-25T19:32:43.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.851+0000 7f7fa6925640 10 monclient: monmap: 2024-03-25T19:32:43.851 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:43.851 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:43.851 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:32:43.852790+0000 2024-03-25T19:32:43.852 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:32:43.852790+0000 2024-03-25T19:32:43.852 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:43.852 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:43.852 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:43.852 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:32:43.852 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:32:43.852 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:43.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.852+0000 7f7fa6925640 1 Processor -- start 2024-03-25T19:32:43.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.852+0000 7f7fa6925640 1 -- start start 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.852+0000 7f7fa6925640 10 monclient: init 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa00a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa013f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa01430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa0142520 con 0x7f7fa01430c0 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa00b28a0 con 0x7f7fa013f710 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa00b2a30 con 0x7f7fa00a98f0 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:43.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.853+0000 7f7fa6925640 10 monclient(hunting): authenticate will time out at 6180.197181s 2024-03-25T19:32:43.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.855+0000 7f7fa4ec2640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f7fa01430c0 0x7f7fa014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:45844/0 (socket says 172.21.15.143:45844) 2024-03-25T19:32:43.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.855+0000 7f7fa4ec2640 1 -- 172.21.15.143:0/2912578741 learned_addr learned my addr 172.21.15.143:0/2912578741 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:43.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.873+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208074636 0 0) 0x7f7fa0142520 con 0x7f7fa01430c0 2024-03-25T19:32:43.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.873+0000 7f7f9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:43.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.873+0000 7f7f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:43.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.873+0000 7f7f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:43.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.873+0000 7f7f9effd640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:32:43.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.873+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f88003520 con 0x7f7fa01430c0 2024-03-25T19:32:43.885 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.885+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54649154 0 0) 0x7f7fa00b2a30 con 0x7f7fa00a98f0 2024-03-25T19:32:43.885 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.885+0000 7f7f9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:43.885 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.885+0000 7f7f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:43.885 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.885+0000 7f7f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:43.885 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.885+0000 7f7f9effd640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:32:43.885 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.885+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa0142520 con 0x7f7fa00a98f0 2024-03-25T19:32:43.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.889+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576867558 0 0) 0x7f7fa00b28a0 con 0x7f7fa013f710 2024-03-25T19:32:43.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.889+0000 7f7f9effd640 10 monclient(hunting): _negotiate 2024-03-25T19:32:43.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.889+0000 7f7f9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:43.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.889+0000 7f7f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:43.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.889+0000 7f7f9effd640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:32:43.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.889+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa00b2a30 con 0x7f7fa013f710 2024-03-25T19:32:43.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3840555331 0 0) 0x7f7f88003520 con 0x7f7fa01430c0 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 >> v1:172.21.15.143:6790/0 conn(0x7f7fa00a98f0 legacy=0x7f7fa00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 >> v1:172.21.15.203:6789/0 conn(0x7f7fa013f710 legacy=0x7f7fa01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 10 monclient: _finish_hunting 0 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 1 monclient: found mon.noname-a 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa00b44c0 con 0x7f7fa01430c0 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 10 monclient: _finish_auth 0 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 10 monclient: _check_auth_tickets 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7f98001a10 con 0x7f7fa01430c0 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7fa6925640 5 monclient: authenticate success, global_id 4344 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7fa6925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:43.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.892+0000 7f7f9effd640 10 monclient: dump: 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:43.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.893+0000 7f7fa6925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.903+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f98001b70 con 0x7f7fa01430c0 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.903+0000 7f7f9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.903+0000 7f7f9effd640 1 -- 172.21.15.143:0/2912578741 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7f98001f80 con 0x7f7fa01430c0 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.903+0000 7f7fa6925640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.903+0000 7f7fa6925640 4 set_mon_vals no callback set 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.903+0000 7f7f9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.903+0000 7f7f9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.903+0000 7f7f9effd640 10 monclient: dump: 2024-03-25T19:32:43.903 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:43.904 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.904+0000 7f7fa6925640 10 monclient: shutdown 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.904+0000 7f7fa6925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.904+0000 7f7fa6925640 1 -- 172.21.15.143:0/2912578741 >> v1:172.21.15.143:6789/0 conn(0x7f7fa01430c0 legacy=0x7f7fa014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.904+0000 7f7fa6925640 1 -- 172.21.15.143:0/2912578741 shutdown_connections 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.904+0000 7f7fa6925640 1 -- 172.21.15.143:0/2912578741 >> 172.21.15.143:0/2912578741 conn(0x7f7fa00b7ea0 msgr2=0x7f7fa00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.904+0000 7f7fa6925640 1 -- 172.21.15.143:0/2912578741 shutdown_connections 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.905+0000 7f7fa6925640 1 -- 172.21.15.143:0/2912578741 wait complete. 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.905+0000 7f7fa6925640 10 monclient: build_initial_monmap 2024-03-25T19:32:43.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.905+0000 7f7fa6925640 10 monclient: monmap: 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:32:43.906 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:43.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.906+0000 7f7fa6925640 1 librados: starting msgr at 2024-03-25T19:32:43.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.906+0000 7f7fa6925640 1 librados: starting objecter 2024-03-25T19:32:43.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.906+0000 7f7fa6925640 1 Processor -- start 2024-03-25T19:32:43.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.906+0000 7f7fa6925640 1 -- start start 2024-03-25T19:32:43.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.906+0000 7f7fa6925640 1 librados: setting wanted keys 2024-03-25T19:32:43.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.906+0000 7f7fa6925640 1 librados: calling monclient init 2024-03-25T19:32:43.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.906+0000 7f7fa6925640 10 monclient: init 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.907+0000 7f7fa6925640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.907+0000 7f7fa6925640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.907+0000 7f7fa6925640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa00a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.907+0000 7f7fa6925640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa013f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.908+0000 7f7fa6925640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa01430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.908+0000 7f7fa6925640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa01d7df0 con 0x7f7fa01430c0 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.908+0000 7f7fa6925640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa01d7f80 con 0x7f7fa013f710 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.908+0000 7f7fa6925640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa013b5c0 con 0x7f7fa00a98f0 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.908+0000 7f7fa6925640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.908+0000 7f7fa6925640 10 monclient(hunting): authenticate will time out at 6180.252025s 2024-03-25T19:32:43.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.915+0000 7f7f9ffff640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f7fa00a98f0 0x7f7fa01d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:56780/0 (socket says 172.21.15.143:56780) 2024-03-25T19:32:43.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.915+0000 7f7f9ffff640 1 -- 172.21.15.143:0/2076330339 learned_addr learned my addr 172.21.15.143:0/2076330339 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:43.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.917+0000 7f7f9cff9640 10 client.?.objecter ms_handle_connect 0x7f7fa00a98f0 2024-03-25T19:32:43.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.917+0000 7f7f9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:43.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.927+0000 7f7f9cff9640 10 client.?.objecter ms_handle_connect 0x7f7fa01430c0 2024-03-25T19:32:43.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.927+0000 7f7f9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:43.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.932+0000 7f7f9cff9640 10 client.?.objecter ms_handle_connect 0x7f7fa013f710 2024-03-25T19:32:43.932 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.932+0000 7f7f9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:43.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.935+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023081779 0 0) 0x7f7fa01d7df0 con 0x7f7fa01430c0 2024-03-25T19:32:43.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.935+0000 7f7f9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:43.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.935+0000 7f7f9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:43.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.935+0000 7f7f9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:43.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.935+0000 7f7f9cff9640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:32:43.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.935+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f8c004eb0 con 0x7f7fa01430c0 2024-03-25T19:32:43.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.937+0000 7f7fa4ec2640 1 -- 172.21.15.143:0/2076330339 >> v1:172.21.15.143:6789/0 conn(0x7f7fa01430c0 legacy=0x7f7fa0135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:32:43.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.937+0000 7f7fa4ec2640 1 -- 172.21.15.143:0/2076330339 >> v1:172.21.15.143:6789/0 conn(0x7f7fa01430c0 legacy=0x7f7fa0135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:32:43.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.937+0000 7f7fa4ec2640 1 --1- 172.21.15.143:0/2076330339 >> v1:172.21.15.143:6789/0 conn(0x7f7fa01430c0 0x7f7fa0135090 :-1 s=OPENED pgs=16714 cs=1 l=1).handle_message read tag failed 2024-03-25T19:32:43.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.937+0000 7f7fa4ec2640 1 --1- 172.21.15.143:0/2076330339 >> v1:172.21.15.143:6789/0 conn(0x7f7fa01430c0 0x7f7fa0135090 :-1 s=OPENED pgs=16714 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:32:43.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.937+0000 7f7f9cff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6789/0 2024-03-25T19:32:43.939 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.940+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483350474 0 0) 0x7f7fa013b5c0 con 0x7f7fa00a98f0 2024-03-25T19:32:43.939 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.940+0000 7f7f9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:43.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.940+0000 7f7f9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:43.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.940+0000 7f7f9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:43.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.940+0000 7f7f9cff9640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:32:43.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.940+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa01d7df0 con 0x7f7fa00a98f0 2024-03-25T19:32:43.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700954340 0 0) 0x7f7fa01d7f80 con 0x7f7fa013f710 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa013b5c0 con 0x7f7fa013f710 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 878956162 0 0) 0x7f7fa01d7df0 con 0x7f7fa00a98f0 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fa01d7f80 con 0x7f7fa00a98f0 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7f90001970 con 0x7f7fa00a98f0 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 10 monclient(hunting): dump: 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:43.946 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:43.947 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.946+0000 7f7f9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:43.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1453935347 0 0) 0x7f7fa01d7f80 con 0x7f7fa00a98f0 2024-03-25T19:32:43.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 >> v1:172.21.15.203:6789/0 conn(0x7f7fa013f710 legacy=0x7f7fa0133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 >> v1:172.21.15.143:6789/0 conn(0x7f7fa01430c0 legacy=0x7f7fa0135090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 1 monclient: found mon.c 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa013b750 con 0x7f7fa00a98f0 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 10 monclient: _finish_auth 0 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7f9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7fa6925640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7fa6925640 10 monclient: _renew_subs 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7fa6925640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa0133710 con 0x7f7fa00a98f0 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7fa6925640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:43.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7fa6925640 10 monclient: _renew_subs 2024-03-25T19:32:43.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.957+0000 7f7fa6925640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:43.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.958+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa01357a0 con 0x7f7fa00a98f0 2024-03-25T19:32:43.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.958+0000 7f7fa6925640 1 librados: init done 2024-03-25T19:32:43.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.958+0000 7f7fa6925640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:43.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.958+0000 7f7fa6925640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:43.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.958+0000 7f7fa6925640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:43.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:43.958+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0138fd0 con 0x7f7fa00a98f0 2024-03-25T19:32:44.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.259+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f90003610 con 0x7f7fa00a98f0 2024-03-25T19:32:44.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.259+0000 7f7f9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:44.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.259+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7f90003840 con 0x7f7fa00a98f0 2024-03-25T19:32:44.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.259+0000 7f7f9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:44.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.260+0000 7f7f9cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:44.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.260+0000 7f7f9cff9640 10 monclient: dump: 2024-03-25T19:32:44.259 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:44.259 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.260+0000 7f7f9e7fc640 4 set_mon_vals no callback set 2024-03-25T19:32:44.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.261+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f7f90003a60 con 0x7f7fa00a98f0 2024-03-25T19:32:44.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.261+0000 7f7f9f7fe640 10 monclient: get_auth_request con 0x7f7f8c06de00 auth_method 0 2024-03-25T19:32:44.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.262+0000 7f7f9cff9640 10 client.4340.objecter ms_handle_connect 0x7f7f8c06de00 2024-03-25T19:32:44.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.266+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f7f90004460 con 0x7f7fa00a98f0 2024-03-25T19:32:44.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.266+0000 7f7f9cff9640 10 client.4340.objecter ms_dispatch 0x7f7fa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:44.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.266+0000 7f7f9cff9640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:44.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.266+0000 7f7f9cff9640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:44.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.266+0000 7f7f9cff9640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T19:32:44.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.266+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f90050110 con 0x7f7fa00a98f0 2024-03-25T19:32:44.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.266+0000 7f7f9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:44.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.266+0000 7f7f9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:44.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.518+0000 7f7fa6925640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:44.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.518+0000 7f7fa6925640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:44.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.518+0000 7f7fa6925640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:32:44.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.518+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fa01d8110 con 0x7f7fa00a98f0 2024-03-25T19:32:44.544 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.544+0000 7f7f9cff9640 1 -- 172.21.15.143:0/2076330339 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f7f90052040 con 0x7f7fa00a98f0 2024-03-25T19:32:44.544 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.544+0000 7f7f9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:44.544 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.544+0000 7f7f9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:44.544 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:32:44.544 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.546+0000 7f7fa6925640 10 librados: watch_flush enter 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.546+0000 7f7fa6925640 10 librados: watch_flush exit 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.546+0000 7f7fa6925640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.546+0000 7f7fa6925640 10 client.4340.objecter successfully canceled tick 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 >> v1:172.21.15.143:6800/1292282666 conn(0x7f7f8c06de00 legacy=0x7f7f8c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 10 monclient: shutdown 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 >> v1:172.21.15.143:6790/0 conn(0x7f7fa00a98f0 legacy=0x7f7fa01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 shutdown_connections 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 >> 172.21.15.143:0/2076330339 conn(0x7f7fa00b7ea0 msgr2=0x7f7fa0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 shutdown_connections 2024-03-25T19:32:44.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 1 -- 172.21.15.143:0/2076330339 wait complete. 2024-03-25T19:32:44.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:44.547+0000 7f7fa6925640 1 librados: shutdown 2024-03-25T19:32:44.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:51.366 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:51.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.484+0000 7f46ed90e640 10 monclient: get_monmap_and_config 2024-03-25T19:32:51.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.484+0000 7f46ed90e640 10 monclient: build_initial_monmap 2024-03-25T19:32:51.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.484+0000 7f46ed90e640 10 monclient: monmap: 2024-03-25T19:32:51.485 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:32:51.486808+0000 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:32:51.486808+0000 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:32:51.486 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:51.487 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.485+0000 7f46ed90e640 1 Processor -- start 2024-03-25T19:32:51.487 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.485+0000 7f46ed90e640 1 -- start start 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.485+0000 7f46ed90e640 10 monclient: init 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 10 monclient(hunting): picked mon.noname-b con 0x7f46e80a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 10 monclient(hunting): picked mon.noname-c con 0x7f46e813f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 10 monclient(hunting): picked mon.noname-a con 0x7f46e81430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e8142520 con 0x7f46e81430c0 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80b28a0 con 0x7f46e813f710 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80b2a30 con 0x7f46e80a98f0 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:51.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.486+0000 7f46ed90e640 10 monclient(hunting): authenticate will time out at 6187.831208s 2024-03-25T19:32:51.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.494+0000 7f46e77fe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f46e81430c0 0x7f46e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:39638/0 (socket says 172.21.15.143:39638) 2024-03-25T19:32:51.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.494+0000 7f46e77fe640 1 -- 172.21.15.143:0/2913005346 learned_addr learned my addr 172.21.15.143:0/2913005346 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:51.509 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.508+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013460434 0 0) 0x7f46e8142520 con 0x7f46e81430c0 2024-03-25T19:32:51.509 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.508+0000 7f46e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:51.509 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.508+0000 7f46e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:51.509 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.508+0000 7f46e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:51.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.508+0000 7f46e5ffb640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:32:51.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.509+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d0003520 con 0x7f46e81430c0 2024-03-25T19:32:51.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.512+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661317063 0 0) 0x7f46e80b28a0 con 0x7f46e813f710 2024-03-25T19:32:51.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.512+0000 7f46e5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:32:51.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.512+0000 7f46e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:51.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.512+0000 7f46e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:51.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.512+0000 7f46e5ffb640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:32:51.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.512+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46e8142520 con 0x7f46e813f710 2024-03-25T19:32:51.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305582537 0 0) 0x7f46d0003520 con 0x7f46e81430c0 2024-03-25T19:32:51.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 >> v1:172.21.15.143:6790/0 conn(0x7f46e80a98f0 legacy=0x7f46e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:51.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 >> v1:172.21.15.203:6789/0 conn(0x7f46e813f710 legacy=0x7f46e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:51.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:32:51.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 1 monclient: found mon.noname-a 2024-03-25T19:32:51.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:51.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e80b44c0 con 0x7f46e81430c0 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f46e0001a10 con 0x7f46e81430c0 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46e5ffb640 10 monclient: dump: 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:51.528 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:51.529 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:51.529 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:51.529 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:51.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46ed90e640 5 monclient: authenticate success, global_id 4350 2024-03-25T19:32:51.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.526+0000 7f46ed90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.532+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46e0001b70 con 0x7f46e81430c0 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.532+0000 7f46e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.532+0000 7f46e5ffb640 1 -- 172.21.15.143:0/2913005346 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f46e0001f80 con 0x7f46e81430c0 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.532+0000 7f46ed90e640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.532+0000 7f46ed90e640 4 set_mon_vals no callback set 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.532+0000 7f46e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.532+0000 7f46e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.532+0000 7f46e5ffb640 10 monclient: dump: 2024-03-25T19:32:51.532 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:51.533 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 10 monclient: shutdown 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 1 -- 172.21.15.143:0/2913005346 >> v1:172.21.15.143:6789/0 conn(0x7f46e81430c0 legacy=0x7f46e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 1 -- 172.21.15.143:0/2913005346 shutdown_connections 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 1 -- 172.21.15.143:0/2913005346 >> 172.21.15.143:0/2913005346 conn(0x7f46e80b7ea0 msgr2=0x7f46e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 1 -- 172.21.15.143:0/2913005346 shutdown_connections 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 1 -- 172.21.15.143:0/2913005346 wait complete. 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 10 monclient: build_initial_monmap 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 10 monclient: monmap: 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:51.536 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 1 librados: starting msgr at 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 1 librados: starting objecter 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.533+0000 7f46ed90e640 1 Processor -- start 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 1 -- start start 2024-03-25T19:32:51.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 1 librados: setting wanted keys 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 1 librados: calling monclient init 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 10 monclient: init 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 10 monclient(hunting): picked mon.noname-b con 0x7f46e80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 10 monclient(hunting): picked mon.noname-a con 0x7f46e813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 10 monclient(hunting): picked mon.noname-c con 0x7f46e81430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e81d7df0 con 0x7f46e813f710 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e81d7f80 con 0x7f46e80a98f0 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e813b5c0 con 0x7f46e81430c0 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.535+0000 7f46ed90e640 10 monclient(hunting): authenticate will time out at 6187.879371s 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.536+0000 7f46e6ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f46e80a98f0 0x7f46e81d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:53992/0 (socket says 172.21.15.143:53992) 2024-03-25T19:32:51.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.536+0000 7f46e6ffd640 1 -- 172.21.15.143:0/1569696710 learned_addr learned my addr 172.21.15.143:0/1569696710 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:51.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.536+0000 7f46cffff640 10 client.?.objecter ms_handle_connect 0x7f46e80a98f0 2024-03-25T19:32:51.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.536+0000 7f46cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:51.540 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.539+0000 7f46cffff640 10 client.?.objecter ms_handle_connect 0x7f46e81430c0 2024-03-25T19:32:51.540 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.539+0000 7f46cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:51.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.605+0000 7f46cffff640 10 client.?.objecter ms_handle_connect 0x7f46e813f710 2024-03-25T19:32:51.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.605+0000 7f46cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:51.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.615+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632119681 0 0) 0x7f46e81d7df0 con 0x7f46e813f710 2024-03-25T19:32:51.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.615+0000 7f46cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:51.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.615+0000 7f46cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:51.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.615+0000 7f46cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:51.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.615+0000 7f46cffff640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:32:51.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.615+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d4004eb0 con 0x7f46e813f710 2024-03-25T19:32:51.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.634+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2692313718 0 0) 0x7f46d4004eb0 con 0x7f46e813f710 2024-03-25T19:32:51.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.634+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46e81d7df0 con 0x7f46e813f710 2024-03-25T19:32:51.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.634+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f46dc001930 con 0x7f46e813f710 2024-03-25T19:32:51.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.634+0000 7f46cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:51.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.634+0000 7f46cffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.635+0000 7f46cffff640 10 monclient(hunting): dump: 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:51.635 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.635+0000 7f46cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3409068207 0 0) 0x7f46e81d7df0 con 0x7f46e813f710 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 >> v1:172.21.15.143:6790/0 conn(0x7f46e81430c0 legacy=0x7f46e8135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 >> v1:172.21.15.203:6789/0 conn(0x7f46e80a98f0 legacy=0x7f46e81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 1 monclient: found mon.a 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e813b750 con 0x7f46e813f710 2024-03-25T19:32:51.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 10 monclient: _finish_auth 0 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 5 monclient: authenticate success, global_id 4353 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 10 monclient: _renew_subs 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e8133710 con 0x7f46e813f710 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 10 monclient: _renew_subs 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e81357a0 con 0x7f46e813f710 2024-03-25T19:32:51.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.646+0000 7f46ed90e640 1 librados: init done 2024-03-25T19:32:51.648 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.647+0000 7f46ed90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:51.648 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.647+0000 7f46ed90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:51.648 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.647+0000 7f46ed90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:51.648 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.647+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e8138fd0 con 0x7f46e813f710 2024-03-25T19:32:51.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.936+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46dc003610 con 0x7f46e813f710 2024-03-25T19:32:51.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.936+0000 7f46cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.936+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f46dc003780 con 0x7f46e813f710 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.936+0000 7f46cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.936+0000 7f46cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.936+0000 7f46cffff640 10 monclient: dump: 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:51.937 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.936+0000 7f46e57fa640 4 set_mon_vals no callback set 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.937+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f46dc019790 con 0x7f46e813f710 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.938+0000 7f46e6ffd640 10 monclient: get_auth_request con 0x7f46d406dc00 auth_method 0 2024-03-25T19:32:51.938 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.938+0000 7f46cffff640 10 client.4353.objecter ms_handle_connect 0x7f46d406dc00 2024-03-25T19:32:51.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.944+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f46dc0534a0 con 0x7f46e813f710 2024-03-25T19:32:51.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.944+0000 7f46cffff640 10 client.4353.objecter ms_dispatch 0x7f46e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:51.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.944+0000 7f46cffff640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:51.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.944+0000 7f46cffff640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:51.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.944+0000 7f46cffff640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T19:32:51.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.944+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46dc04ede0 con 0x7f46e813f710 2024-03-25T19:32:51.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.944+0000 7f46cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:51.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:51.944+0000 7f46cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:52.190 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.190+0000 7f46ed90e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:52.190 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.190+0000 7f46ed90e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:52.190 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.190+0000 7f46ed90e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:32:52.190 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.190+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46e813b8e0 con 0x7f46e813f710 2024-03-25T19:32:52.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.233+0000 7f46cffff640 1 -- 172.21.15.143:0/1569696710 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f46e813b8e0 con 0x7f46e813f710 2024-03-25T19:32:52.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.233+0000 7f46cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:52.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.233+0000 7f46cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:52.233 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:32:52.233 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:52.235 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 10 librados: watch_flush enter 2024-03-25T19:32:52.235 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 10 librados: watch_flush exit 2024-03-25T19:32:52.235 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 10 client.4353.objecter successfully canceled tick 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 >> v1:172.21.15.143:6800/1292282666 conn(0x7f46d406dc00 legacy=0x7f46d40700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 10 monclient: shutdown 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 >> v1:172.21.15.143:6789/0 conn(0x7f46e813f710 legacy=0x7f46e8133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 shutdown_connections 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.235+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 >> 172.21.15.143:0/1569696710 conn(0x7f46e80b7ea0 msgr2=0x7f46e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.236+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 shutdown_connections 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.236+0000 7f46ed90e640 1 -- 172.21.15.143:0/1569696710 wait complete. 2024-03-25T19:32:52.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:52.236+0000 7f46ed90e640 1 librados: shutdown 2024-03-25T19:32:52.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:32:53.231 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:19.796 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:53.249 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:19.814 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:53.275 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:19.811 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:53.282 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:19.847 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:53.291 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:19.827 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:53.300 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:03:19.865 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:53.307 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:03:19.872 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:53.315 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:19.851 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:53.330 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:03:19.895 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:59.055 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:32:59.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.165+0000 7f2425dcf640 10 monclient: get_monmap_and_config 2024-03-25T19:32:59.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.165+0000 7f2425dcf640 10 monclient: build_initial_monmap 2024-03-25T19:32:59.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.165+0000 7f2425dcf640 10 monclient: monmap: 2024-03-25T19:32:59.165 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:59.165 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:59.165 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:32:59.166671+0000 2024-03-25T19:32:59.165 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:32:59.166671+0000 2024-03-25T19:32:59.165 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:59.166 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:59.166 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:59.166 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:32:59.166 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:32:59.166 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:59.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.166+0000 7f2425dcf640 1 Processor -- start 2024-03-25T19:32:59.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.166+0000 7f2425dcf640 1 -- start start 2024-03-25T19:32:59.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.166+0000 7f2425dcf640 10 monclient: init 2024-03-25T19:32:59.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:59.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:59.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f24200a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:59.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f242013f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:59.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f24201430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:59.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2420142520 con 0x7f24201430c0 2024-03-25T19:32:59.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24200b28a0 con 0x7f24200a98f0 2024-03-25T19:32:59.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24200b2a30 con 0x7f242013f710 2024-03-25T19:32:59.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:59.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.167+0000 7f2425dcf640 10 monclient(hunting): authenticate will time out at 6195.511070s 2024-03-25T19:32:59.172 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.173+0000 7f241f7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f24200a98f0 0x7f24200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:57562/0 (socket says 172.21.15.143:57562) 2024-03-25T19:32:59.172 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.173+0000 7f241f7fe640 1 -- 172.21.15.143:0/3848660256 learned_addr learned my addr 172.21.15.143:0/3848660256 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:59.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.185+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597409615 0 0) 0x7f24200b28a0 con 0x7f24200a98f0 2024-03-25T19:32:59.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.185+0000 7f241e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.185+0000 7f241e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.185+0000 7f241e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.185+0000 7f241e7fc640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:32:59.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.185+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2408003520 con 0x7f24200a98f0 2024-03-25T19:32:59.186 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.187+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728365626 0 0) 0x7f24200b2a30 con 0x7f242013f710 2024-03-25T19:32:59.187 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.187+0000 7f241e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.187 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.187+0000 7f241e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.187 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.187+0000 7f241e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.187 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.187+0000 7f241e7fc640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:32:59.187 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.187+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24200b28a0 con 0x7f242013f710 2024-03-25T19:32:59.193 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2931560316 0 0) 0x7f24200b28a0 con 0x7f242013f710 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 >> v1:172.21.15.203:6789/0 conn(0x7f24200a98f0 legacy=0x7f24200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 >> v1:172.21.15.143:6789/0 conn(0x7f24201430c0 legacy=0x7f242014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24200b44c0 con 0x7f242013f710 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2414001670 con 0x7f242013f710 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:59.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f241e7fc640 10 monclient: dump: 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f2425dcf640 5 monclient: authenticate success, global_id 4346 2024-03-25T19:32:59.195 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.194+0000 7f2425dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:32:59.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.201+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24140017d0 con 0x7f242013f710 2024-03-25T19:32:59.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.201+0000 7f241e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:59.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.201+0000 7f2425dcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:32:59.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.201+0000 7f2425dcf640 4 set_mon_vals no callback set 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f241e7fc640 1 -- 172.21.15.143:0/3848660256 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2414001b60 con 0x7f242013f710 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f241e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f241e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f241e7fc640 10 monclient: dump: 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.202 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f2425dcf640 10 monclient: shutdown 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f2425dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f2425dcf640 1 -- 172.21.15.143:0/3848660256 >> v1:172.21.15.143:6790/0 conn(0x7f242013f710 legacy=0x7f24201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f2425dcf640 1 -- 172.21.15.143:0/3848660256 shutdown_connections 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f2425dcf640 1 -- 172.21.15.143:0/3848660256 >> 172.21.15.143:0/3848660256 conn(0x7f24200b7ea0 msgr2=0x7f24200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f2425dcf640 1 -- 172.21.15.143:0/3848660256 shutdown_connections 2024-03-25T19:32:59.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.202+0000 7f2425dcf640 1 -- 172.21.15.143:0/3848660256 wait complete. 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.203+0000 7f2425dcf640 10 monclient: build_initial_monmap 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.203+0000 7f2425dcf640 10 monclient: monmap: 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:32:59.205 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.203+0000 7f2425dcf640 1 librados: starting msgr at 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.203+0000 7f2425dcf640 1 librados: starting objecter 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 1 Processor -- start 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 1 -- start start 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 1 librados: setting wanted keys 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 1 librados: calling monclient init 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 10 monclient: init 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f24200a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f242013f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f24201430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:32:59.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24201daee0 con 0x7f24201430c0 2024-03-25T19:32:59.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24201dba80 con 0x7f24200a98f0 2024-03-25T19:32:59.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24200aba30 con 0x7f242013f710 2024-03-25T19:32:59.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:32:59.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.204+0000 7f2425dcf640 10 monclient(hunting): authenticate will time out at 6195.548610s 2024-03-25T19:32:59.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.211+0000 7f241f7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f24200a98f0 0x7f24201db370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:57598/0 (socket says 172.21.15.143:57598) 2024-03-25T19:32:59.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.211+0000 7f241f7fe640 1 -- 172.21.15.143:0/3712634714 learned_addr learned my addr 172.21.15.143:0/3712634714 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:32:59.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.215+0000 7f2407fff640 10 client.?.objecter ms_handle_connect 0x7f24200a98f0 2024-03-25T19:32:59.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.215+0000 7f2407fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:59.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.217+0000 7f2407fff640 10 client.?.objecter ms_handle_connect 0x7f24201430c0 2024-03-25T19:32:59.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.217+0000 7f2407fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:59.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.232+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164751205 0 0) 0x7f24201dba80 con 0x7f24200a98f0 2024-03-25T19:32:59.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.232+0000 7f2407fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.232+0000 7f2407fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.232+0000 7f2407fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.232+0000 7f2407fff640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:32:59.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.232+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c004eb0 con 0x7f24200a98f0 2024-03-25T19:32:59.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.238+0000 7f2407fff640 10 client.?.objecter ms_handle_connect 0x7f242013f710 2024-03-25T19:32:59.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.238+0000 7f2407fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:32:59.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.239+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487556297 0 0) 0x7f24201daee0 con 0x7f24201430c0 2024-03-25T19:32:59.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.240+0000 7f2407fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.240+0000 7f2407fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.240+0000 7f2407fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.240+0000 7f2407fff640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:32:59.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.240+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24201dba80 con 0x7f24201430c0 2024-03-25T19:32:59.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.245+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 584069753 0 0) 0x7f240c004eb0 con 0x7f24200a98f0 2024-03-25T19:32:59.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.245+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24201daee0 con 0x7f24200a98f0 2024-03-25T19:32:59.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.245+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2410001950 con 0x7f24200a98f0 2024-03-25T19:32:59.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.245+0000 7f2407fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.245+0000 7f2407fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:32:59.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.245+0000 7f2407fff640 10 monclient(hunting): dump: 2024-03-25T19:32:59.245 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:59.245 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:59.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.245+0000 7f2407fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:32:59.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.257+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021810030 0 0) 0x7f24200aba30 con 0x7f242013f710 2024-03-25T19:32:59.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.257+0000 7f2407fff640 10 monclient(hunting): _negotiate 2024-03-25T19:32:59.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.257+0000 7f2407fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:32:59.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.257+0000 7f2407fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:32:59.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.257+0000 7f2407fff640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:32:59.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.257+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f240c004eb0 con 0x7f242013f710 2024-03-25T19:32:59.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.270+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1249145411 0 0) 0x7f24201dba80 con 0x7f24201430c0 2024-03-25T19:32:59.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.270+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24200aba30 con 0x7f24201430c0 2024-03-25T19:32:59.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.271+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2418001b10 con 0x7f24201430c0 2024-03-25T19:32:59.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.271+0000 7f2407fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.271+0000 7f2407fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.271+0000 7f2407fff640 10 monclient(hunting): dump: 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:59.271 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:59.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1458553853 0 0) 0x7f24201daee0 con 0x7f24200a98f0 2024-03-25T19:32:59.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 >> v1:172.21.15.143:6790/0 conn(0x7f242013f710 legacy=0x7f2420138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 >> v1:172.21.15.143:6789/0 conn(0x7f24201430c0 legacy=0x7f24201e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 10 monclient: _finish_hunting 0 2024-03-25T19:32:59.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 1 monclient: found mon.b 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24201dab70 con 0x7f24200a98f0 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 10 monclient: _finish_auth 0 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 10 monclient: _check_auth_tickets 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2407fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 5 monclient: authenticate success, global_id 4363 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 10 monclient: _renew_subs 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24201d51f0 con 0x7f24200a98f0 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 10 monclient: _renew_subs 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:32:59.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2420138c40 con 0x7f24200a98f0 2024-03-25T19:32:59.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.277+0000 7f2425dcf640 1 librados: init done 2024-03-25T19:32:59.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.278+0000 7f2425dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:59.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.278+0000 7f2425dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:59.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.278+0000 7f2425dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:32:59.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.278+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24200627b0 con 0x7f24200a98f0 2024-03-25T19:32:59.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.288+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24100047e0 con 0x7f24200a98f0 2024-03-25T19:32:59.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.289+0000 7f2407fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:32:59.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.289+0000 7f241dffb640 4 set_mon_vals no callback set 2024-03-25T19:32:59.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.567+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2410001d80 con 0x7f24200a98f0 2024-03-25T19:32:59.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.568+0000 7f2407fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:32:59.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.568+0000 7f2407fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:32:59.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.568+0000 7f2407fff640 10 monclient: dump: 2024-03-25T19:32:59.567 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:32:59.568 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:32:59.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.568+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f241001aa60 con 0x7f24200a98f0 2024-03-25T19:32:59.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.569+0000 7f241effd640 10 monclient: get_auth_request con 0x7f240c06dec0 auth_method 0 2024-03-25T19:32:59.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.569+0000 7f2407fff640 10 client.4363.objecter ms_handle_connect 0x7f240c06dec0 2024-03-25T19:32:59.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.575+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f24100548d0 con 0x7f24200a98f0 2024-03-25T19:32:59.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.575+0000 7f2407fff640 10 client.4363.objecter ms_dispatch 0x7f2420000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:32:59.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.575+0000 7f2407fff640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:32:59.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.575+0000 7f2407fff640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:32:59.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.575+0000 7f2407fff640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.578+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2410050210 con 0x7f24200a98f0 2024-03-25T19:32:59.577 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.578+0000 7f2407fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:32:59.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.578+0000 7f2407fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:32:59.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.806+0000 7f2425dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:59.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.806+0000 7f2425dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:59.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.806+0000 7f2425dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:32:59.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.806+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2420074cc0 con 0x7f24200a98f0 2024-03-25T19:32:59.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.851+0000 7f2407fff640 1 -- 172.21.15.143:0/3712634714 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f2410052310 con 0x7f24200a98f0 2024-03-25T19:32:59.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.851+0000 7f2407fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:32:59.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.851+0000 7f2407fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:32:59.851 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:32:59.851 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:32:59.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 10 librados: watch_flush enter 2024-03-25T19:32:59.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 10 librados: watch_flush exit 2024-03-25T19:32:59.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T19:32:59.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 10 client.4363.objecter successfully canceled tick 2024-03-25T19:32:59.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 >> v1:172.21.15.143:6800/1292282666 conn(0x7f240c06dec0 legacy=0x7f240c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 10 monclient: shutdown 2024-03-25T19:32:59.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:32:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 >> v1:172.21.15.203:6789/0 conn(0x7f24200a98f0 legacy=0x7f24201db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:32:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 shutdown_connections 2024-03-25T19:32:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 >> 172.21.15.143:0/3712634714 conn(0x7f24200b7ea0 msgr2=0x7f2420074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:32:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 shutdown_connections 2024-03-25T19:32:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 1 -- 172.21.15.143:0/3712634714 wait complete. 2024-03-25T19:32:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:32:59.854+0000 7f2425dcf640 1 librados: shutdown 2024-03-25T19:32:59.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:06.674 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:06.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.795+0000 7fc42fab3640 10 monclient: get_monmap_and_config 2024-03-25T19:33:06.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.795+0000 7fc42fab3640 10 monclient: build_initial_monmap 2024-03-25T19:33:06.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.795+0000 7fc42fab3640 10 monclient: monmap: 2024-03-25T19:33:06.795 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:33:06.796788+0000 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:33:06.796788+0000 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:33:06.796 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.796+0000 7fc42fab3640 1 Processor -- start 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.796+0000 7fc42fab3640 1 -- start start 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.796+0000 7fc42fab3640 10 monclient: init 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 10 monclient(hunting): picked mon.noname-c con 0x7fc4280b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 10 monclient(hunting): picked mon.noname-b con 0x7fc4280b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 10 monclient(hunting): picked mon.noname-a con 0x7fc42814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280af940 con 0x7fc42814be90 2024-03-25T19:33:06.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280b0a70 con 0x7fc4280b4d60 2024-03-25T19:33:06.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280b0d50 con 0x7fc4280b1730 2024-03-25T19:33:06.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:06.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.797+0000 7fc42fab3640 10 monclient(hunting): authenticate will time out at 6203.141217s 2024-03-25T19:33:06.804 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.804+0000 7fc42d84f640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc4280b4d60 0x7fc42813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:35800/0 (socket says 172.21.15.143:35800) 2024-03-25T19:33:06.804 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.804+0000 7fc42d84f640 1 -- 172.21.15.143:0/2348011031 learned_addr learned my addr 172.21.15.143:0/2348011031 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:06.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.811+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631341047 0 0) 0x7fc4280af940 con 0x7fc42814be90 2024-03-25T19:33:06.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.811+0000 7fc42c84d640 10 monclient(hunting): _negotiate 2024-03-25T19:33:06.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.811+0000 7fc42c84d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:06.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.811+0000 7fc42c84d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:06.811 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.811+0000 7fc42c84d640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:33:06.811 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.811+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc414003520 con 0x7fc42814be90 2024-03-25T19:33:06.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297632596 0 0) 0x7fc414003520 con 0x7fc42814be90 2024-03-25T19:33:06.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 >> v1:172.21.15.143:6790/0 conn(0x7fc4280b1730 legacy=0x7fc4280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 >> v1:172.21.15.203:6789/0 conn(0x7fc4280b4d60 legacy=0x7fc42813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 10 monclient: _finish_hunting 0 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 1 monclient: found mon.noname-a 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4280b1030 con 0x7fc42814be90 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 10 monclient: _finish_auth 0 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 10 monclient: _check_auth_tickets 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc420001a10 con 0x7fc42814be90 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42fab3640 5 monclient: authenticate success, global_id 4362 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42c84d640 10 monclient: dump: 2024-03-25T19:33:06.819 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:06.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.818+0000 7fc42fab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.821+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc420001c30 con 0x7fc42814be90 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.821+0000 7fc42c84d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.821+0000 7fc42c84d640 1 -- 172.21.15.143:0/2348011031 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc420001fc0 con 0x7fc42814be90 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.821+0000 7fc42c84d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.821+0000 7fc42c84d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.821+0000 7fc42c84d640 10 monclient: dump: 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:06.821 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.821+0000 7fc42fab3640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:06.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.821+0000 7fc42fab3640 4 set_mon_vals no callback set 2024-03-25T19:33:06.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.822+0000 7fc42fab3640 10 monclient: shutdown 2024-03-25T19:33:06.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.822+0000 7fc42fab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:06.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.822+0000 7fc42fab3640 1 -- 172.21.15.143:0/2348011031 >> v1:172.21.15.143:6789/0 conn(0x7fc42814be90 legacy=0x7fc42814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:06.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.822+0000 7fc42fab3640 1 -- 172.21.15.143:0/2348011031 shutdown_connections 2024-03-25T19:33:06.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.822+0000 7fc42fab3640 1 -- 172.21.15.143:0/2348011031 >> 172.21.15.143:0/2348011031 conn(0x7fc4280617a0 msgr2=0x7fc428061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:06.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.822+0000 7fc42fab3640 1 -- 172.21.15.143:0/2348011031 shutdown_connections 2024-03-25T19:33:06.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.822+0000 7fc42fab3640 1 -- 172.21.15.143:0/2348011031 wait complete. 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 10 monclient: build_initial_monmap 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 10 monclient: monmap: 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 1 librados: starting msgr at 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 1 librados: starting objecter 2024-03-25T19:33:06.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 1 Processor -- start 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 1 -- start start 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 1 librados: setting wanted keys 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 1 librados: calling monclient init 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.823+0000 7fc42fab3640 10 monclient: init 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 10 monclient(hunting): picked mon.noname-c con 0x7fc4280b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 10 monclient(hunting): picked mon.noname-a con 0x7fc4280b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 10 monclient(hunting): picked mon.noname-b con 0x7fc42814be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4281307c0 con 0x7fc4280b4d60 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc428138f90 con 0x7fc42814be90 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc428132ab0 con 0x7fc4280b1730 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:06.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.824+0000 7fc42fab3640 10 monclient(hunting): authenticate will time out at 6203.168222s 2024-03-25T19:33:06.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.833+0000 7fc42e050640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc42814be90 0x7fc4281300b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:35818/0 (socket says 172.21.15.143:35818) 2024-03-25T19:33:06.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.833+0000 7fc42e050640 1 -- 172.21.15.143:0/3379765314 learned_addr learned my addr 172.21.15.143:0/3379765314 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:06.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.833+0000 7fc41e7fc640 10 client.?.objecter ms_handle_connect 0x7fc42814be90 2024-03-25T19:33:06.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.833+0000 7fc41e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:06.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.849+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660091611 0 0) 0x7fc428138f90 con 0x7fc42814be90 2024-03-25T19:33:06.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.849+0000 7fc41e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:06.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.849+0000 7fc41e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:06.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.849+0000 7fc41e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:06.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.849+0000 7fc41e7fc640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:33:06.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.849+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc40c004eb0 con 0x7fc42814be90 2024-03-25T19:33:06.852 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.852+0000 7fc41e7fc640 10 client.?.objecter ms_handle_connect 0x7fc4280b1730 2024-03-25T19:33:06.852 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.852+0000 7fc41e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:06.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.864+0000 7fc41e7fc640 10 client.?.objecter ms_handle_connect 0x7fc4280b4d60 2024-03-25T19:33:06.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.864+0000 7fc41e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:06.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.866+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3559819638 0 0) 0x7fc40c004eb0 con 0x7fc42814be90 2024-03-25T19:33:06.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.866+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc428138f90 con 0x7fc42814be90 2024-03-25T19:33:06.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.866+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc420001f40 con 0x7fc42814be90 2024-03-25T19:33:06.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.866+0000 7fc41e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.866+0000 7fc41e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.866+0000 7fc41e7fc640 10 monclient(hunting): dump: 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:06.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.866+0000 7fc41e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:06.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.868+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19987431 0 0) 0x7fc428132ab0 con 0x7fc4280b1730 2024-03-25T19:33:06.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.868+0000 7fc41e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:06.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.868+0000 7fc41e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:06.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.868+0000 7fc41e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:06.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.868+0000 7fc41e7fc640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:33:06.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.868+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc40c004eb0 con 0x7fc4280b1730 2024-03-25T19:33:06.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2004239436 0 0) 0x7fc428138f90 con 0x7fc42814be90 2024-03-25T19:33:06.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 >> v1:172.21.15.143:6790/0 conn(0x7fc4280b1730 legacy=0x7fc4281393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:06.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 >> v1:172.21.15.143:6789/0 conn(0x7fc4280b4d60 legacy=0x7fc42812f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 1 monclient: found mon.b 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc428133a40 con 0x7fc42814be90 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc41e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 5 monclient: authenticate success, global_id 4369 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 10 monclient: _renew_subs 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4281316c0 con 0x7fc42814be90 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 10 monclient: _renew_subs 2024-03-25T19:33:06.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:06.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc428137860 con 0x7fc42814be90 2024-03-25T19:33:06.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.872+0000 7fc42fab3640 1 librados: init done 2024-03-25T19:33:06.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.874+0000 7fc42fab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:06.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.874+0000 7fc42fab3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:06.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.874+0000 7fc42fab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:06.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:06.874+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc428132c40 con 0x7fc42814be90 2024-03-25T19:33:07.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.163+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc420003c20 con 0x7fc42814be90 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.163+0000 7fc41e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.163+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc420004030 con 0x7fc42814be90 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.163+0000 7fc41e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.163+0000 7fc41e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.163+0000 7fc41e7fc640 10 monclient: dump: 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:07.163 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:07.164 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:07.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.163+0000 7fc41ffff640 4 set_mon_vals no callback set 2024-03-25T19:33:07.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.164+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fc420004250 con 0x7fc42814be90 2024-03-25T19:33:07.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.164+0000 7fc42d84f640 10 monclient: get_auth_request con 0x7fc40c06de20 auth_method 0 2024-03-25T19:33:07.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.165+0000 7fc41e7fc640 10 client.4369.objecter ms_handle_connect 0x7fc40c06de20 2024-03-25T19:33:07.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.169+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fc420054ad0 con 0x7fc42814be90 2024-03-25T19:33:07.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.169+0000 7fc41e7fc640 10 client.4369.objecter ms_dispatch 0x7fc428000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:07.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.169+0000 7fc41e7fc640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:07.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.169+0000 7fc41e7fc640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:07.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.169+0000 7fc41e7fc640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-25T19:33:07.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.170+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4200505e0 con 0x7fc42814be90 2024-03-25T19:33:07.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.170+0000 7fc41e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:07.169 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.170+0000 7fc41e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:07.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.419+0000 7fc42fab3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:07.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.419+0000 7fc42fab3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:07.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.419+0000 7fc42fab3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:07.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.419+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4280627b0 con 0x7fc42814be90 2024-03-25T19:33:07.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.567+0000 7fc41e7fc640 1 -- 172.21.15.143:0/3379765314 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fc420052510 con 0x7fc42814be90 2024-03-25T19:33:07.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.567+0000 7fc41e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:07.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.567+0000 7fc41e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:07.567 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:33:07.567 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 10 librados: watch_flush enter 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 10 librados: watch_flush exit 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 10 client.4369.objecter successfully canceled tick 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 >> v1:172.21.15.143:6800/1292282666 conn(0x7fc40c06de20 legacy=0x7fc40c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 10 monclient: shutdown 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 >> v1:172.21.15.203:6789/0 conn(0x7fc42814be90 legacy=0x7fc4281300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 shutdown_connections 2024-03-25T19:33:07.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 >> 172.21.15.143:0/3379765314 conn(0x7fc4280617a0 msgr2=0x7fc42813c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:07.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 shutdown_connections 2024-03-25T19:33:07.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 1 -- 172.21.15.143:0/3379765314 wait complete. 2024-03-25T19:33:07.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:07.570+0000 7fc42fab3640 1 librados: shutdown 2024-03-25T19:33:07.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:13.312 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:39.877 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:13.318 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:39.882 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:13.355 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:39.920 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:13.366 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:39.902 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:13.374 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:39.910 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:13.395 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:03:39.960 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:13.403 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:39.939 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:13.409 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:03:39.974 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:13.441 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:03:40.006 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:14.388 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:14.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.492+0000 7fa58dcff640 10 monclient: get_monmap_and_config 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.492+0000 7fa58dcff640 10 monclient: build_initial_monmap 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.492+0000 7fa58dcff640 10 monclient: monmap: 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:33:14.493230+0000 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:33:14.493230+0000 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:33:14.492 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:33:14.493 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 1 Processor -- start 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 1 -- start start 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 10 monclient: init 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 10 monclient(hunting): picked mon.noname-a con 0x7fa58806c380 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 10 monclient(hunting): picked mon.noname-b con 0x7fa58806d670 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 10 monclient(hunting): picked mon.noname-c con 0x7fa5880af8f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58806c9a0 con 0x7fa58806c380 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5880ada90 con 0x7fa5880af8f0 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5880b0d10 con 0x7fa58806d670 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:14.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.493+0000 7fa58dcff640 10 monclient(hunting): authenticate will time out at 6210.837335s 2024-03-25T19:33:14.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.496+0000 7fa587fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa5880af8f0 0x7fa5880aedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:40634/0 (socket says 172.21.15.143:40634) 2024-03-25T19:33:14.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.496+0000 7fa587fff640 1 -- 172.21.15.143:0/4288640841 learned_addr learned my addr 172.21.15.143:0/4288640841 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:14.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.517+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022494453 0 0) 0x7fa5880ada90 con 0x7fa5880af8f0 2024-03-25T19:33:14.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.517+0000 7fa5867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.517+0000 7fa5867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.517+0000 7fa5867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.517+0000 7fa5867fc640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:33:14.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.517+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa570003520 con 0x7fa5880af8f0 2024-03-25T19:33:14.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.525+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970900682 0 0) 0x7fa5880b0d10 con 0x7fa58806d670 2024-03-25T19:33:14.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.525+0000 7fa5867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.525+0000 7fa5867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.525+0000 7fa5867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.525+0000 7fa5867fc640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:33:14.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.525+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5880ada90 con 0x7fa58806d670 2024-03-25T19:33:14.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.529+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3216614897 0 0) 0x7fa570003520 con 0x7fa5880af8f0 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.529+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 >> v1:172.21.15.143:6790/0 conn(0x7fa58806d670 legacy=0x7fa5880ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.529+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 >> v1:172.21.15.143:6789/0 conn(0x7fa58806c380 legacy=0x7fa5880b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 1 monclient: found mon.noname-c 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5880add00 con 0x7fa5880af8f0 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa580001a10 con 0x7fa5880af8f0 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa58dcff640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:33:14.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa58dcff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa5867fc640 10 monclient: dump: 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:14.531 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.530+0000 7fa58dcff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa580001b70 con 0x7fa5880af8f0 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa5867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa5867fc640 1 -- 172.21.15.143:0/4288640841 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa580001f00 con 0x7fa5880af8f0 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa58dcff640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa58dcff640 4 set_mon_vals no callback set 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa5867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa5867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa5867fc640 10 monclient: dump: 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:14.538 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.539 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:14.539 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:14.539 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:14.539 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:14.539 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:14.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa58dcff640 10 monclient: shutdown 2024-03-25T19:33:14.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa58dcff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:14.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.538+0000 7fa58dcff640 1 -- 172.21.15.143:0/4288640841 >> v1:172.21.15.203:6789/0 conn(0x7fa5880af8f0 legacy=0x7fa5880aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.539+0000 7fa58dcff640 1 -- 172.21.15.143:0/4288640841 shutdown_connections 2024-03-25T19:33:14.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.539+0000 7fa58dcff640 1 -- 172.21.15.143:0/4288640841 >> 172.21.15.143:0/4288640841 conn(0x7fa5880b5c00 msgr2=0x7fa5880b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:14.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.539+0000 7fa58dcff640 1 -- 172.21.15.143:0/4288640841 shutdown_connections 2024-03-25T19:33:14.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.539+0000 7fa58dcff640 1 -- 172.21.15.143:0/4288640841 wait complete. 2024-03-25T19:33:14.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.539+0000 7fa58dcff640 10 monclient: build_initial_monmap 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.539+0000 7fa58dcff640 10 monclient: monmap: 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.540+0000 7fa58dcff640 1 librados: starting msgr at 2024-03-25T19:33:14.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.540+0000 7fa58dcff640 1 librados: starting objecter 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.540+0000 7fa58dcff640 1 Processor -- start 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.540+0000 7fa58dcff640 1 -- start start 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.540+0000 7fa58dcff640 1 librados: setting wanted keys 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.540+0000 7fa58dcff640 1 librados: calling monclient init 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.540+0000 7fa58dcff640 10 monclient: init 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 10 monclient(hunting): picked mon.noname-c con 0x7fa58806c380 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 10 monclient(hunting): picked mon.noname-b con 0x7fa58806d670 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 10 monclient(hunting): picked mon.noname-a con 0x7fa5880af8f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa588136780 con 0x7fa5880af8f0 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa588144840 con 0x7fa58806d670 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa588144e80 con 0x7fa58806c380 2024-03-25T19:33:14.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:14.544 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.541+0000 7fa58dcff640 10 monclient(hunting): authenticate will time out at 6210.885324s 2024-03-25T19:33:14.552 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.552+0000 7fa586ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa58806d670 0x7fa588135960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:40648/0 (socket says 172.21.15.143:40648) 2024-03-25T19:33:14.552 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.552+0000 7fa586ffd640 1 -- 172.21.15.143:0/2428642352 learned_addr learned my addr 172.21.15.143:0/2428642352 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:14.554 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.555+0000 7fa56ffff640 10 client.?.objecter ms_handle_connect 0x7fa58806d670 2024-03-25T19:33:14.555 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.555+0000 7fa56ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:14.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.562+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515992492 0 0) 0x7fa588144840 con 0x7fa58806d670 2024-03-25T19:33:14.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.562+0000 7fa56ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.562+0000 7fa56ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.562+0000 7fa56ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.562+0000 7fa56ffff640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:33:14.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.563+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa574004eb0 con 0x7fa58806d670 2024-03-25T19:33:14.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.568+0000 7fa56ffff640 10 client.?.objecter ms_handle_connect 0x7fa5880af8f0 2024-03-25T19:33:14.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.568+0000 7fa56ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:14.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.569+0000 7fa56ffff640 10 client.?.objecter ms_handle_connect 0x7fa58806c380 2024-03-25T19:33:14.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.569+0000 7fa56ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:14.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.576+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170028622 0 0) 0x7fa588136780 con 0x7fa5880af8f0 2024-03-25T19:33:14.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.576+0000 7fa56ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.576+0000 7fa56ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.576+0000 7fa56ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.576+0000 7fa56ffff640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:33:14.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.576+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa588144840 con 0x7fa5880af8f0 2024-03-25T19:33:14.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.579+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67266345 0 0) 0x7fa588144e80 con 0x7fa58806c380 2024-03-25T19:33:14.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.579+0000 7fa56ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:33:14.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.579+0000 7fa56ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:14.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.579+0000 7fa56ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:14.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.579+0000 7fa56ffff640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:33:14.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.579+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa588136780 con 0x7fa58806c380 2024-03-25T19:33:14.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.592+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596536242 0 0) 0x7fa574004eb0 con 0x7fa58806d670 2024-03-25T19:33:14.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.592+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa588144e80 con 0x7fa58806d670 2024-03-25T19:33:14.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.592+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa57c0019f0 con 0x7fa58806d670 2024-03-25T19:33:14.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.592+0000 7fa56ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.592+0000 7fa56ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.592+0000 7fa56ffff640 10 monclient(hunting): dump: 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:14.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.592+0000 7fa56ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:14.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1028692078 0 0) 0x7fa588144e80 con 0x7fa58806d670 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 >> v1:172.21.15.143:6790/0 conn(0x7fa58806c380 legacy=0x7fa5881d7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 >> v1:172.21.15.143:6789/0 conn(0x7fa5880af8f0 legacy=0x7fa588136070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 1 monclient: found mon.b 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa588145010 con 0x7fa58806d670 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 10 monclient: _finish_auth 0 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa56ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 5 monclient: authenticate success, global_id 4375 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 10 monclient: _renew_subs 2024-03-25T19:33:14.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:14.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa588147ae0 con 0x7fa58806d670 2024-03-25T19:33:14.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:14.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 10 monclient: _renew_subs 2024-03-25T19:33:14.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:14.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa588144220 con 0x7fa58806d670 2024-03-25T19:33:14.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.603+0000 7fa58dcff640 1 librados: init done 2024-03-25T19:33:14.606 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.605+0000 7fa58dcff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:14.606 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.605+0000 7fa58dcff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:14.606 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.605+0000 7fa58dcff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:14.606 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.605+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58806d1e0 con 0x7fa58806d670 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.892+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa57c001c50 con 0x7fa58806d670 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.892+0000 7fa56ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.892+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa57c003a60 con 0x7fa58806d670 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.892+0000 7fa56ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.892+0000 7fa56ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.892+0000 7fa56ffff640 10 monclient: dump: 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:14.892 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:14.893 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:14.893 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:14.893 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:14.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.892+0000 7fa585ffb640 4 set_mon_vals no callback set 2024-03-25T19:33:14.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.893+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fa57c003c80 con 0x7fa58806d670 2024-03-25T19:33:14.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.893+0000 7fa5877fe640 10 monclient: get_auth_request con 0x7fa57406ddd0 auth_method 0 2024-03-25T19:33:14.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.894+0000 7fa56ffff640 10 client.4375.objecter ms_handle_connect 0x7fa57406ddd0 2024-03-25T19:33:14.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.898+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fa57c0548a0 con 0x7fa58806d670 2024-03-25T19:33:14.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.898+0000 7fa56ffff640 10 client.4375.objecter ms_dispatch 0x7fa588000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:14.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.898+0000 7fa56ffff640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:14.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.898+0000 7fa56ffff640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:14.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.898+0000 7fa56ffff640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T19:33:14.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.898+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa57c0503b0 con 0x7fa58806d670 2024-03-25T19:33:14.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.898+0000 7fa56ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:14.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:14.898+0000 7fa56ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:15.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.145+0000 7fa58dcff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:15.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.145+0000 7fa58dcff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:15.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.145+0000 7fa58dcff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:15.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.145+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5881449d0 con 0x7fa58806d670 2024-03-25T19:33:15.173 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.173+0000 7fa56ffff640 1 -- 172.21.15.143:0/2428642352 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fa57c0522e0 con 0x7fa58806d670 2024-03-25T19:33:15.173 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.173+0000 7fa56ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:15.173 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.173+0000 7fa56ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:15.173 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:33:15.173 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:15.176 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 10 librados: watch_flush enter 2024-03-25T19:33:15.176 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 10 librados: watch_flush exit 2024-03-25T19:33:15.176 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T19:33:15.176 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 10 client.4375.objecter successfully canceled tick 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 >> v1:172.21.15.143:6800/1292282666 conn(0x7fa57406ddd0 legacy=0x7fa574070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 10 monclient: shutdown 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 >> v1:172.21.15.203:6789/0 conn(0x7fa58806d670 legacy=0x7fa588135960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 shutdown_connections 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 >> 172.21.15.143:0/2428642352 conn(0x7fa5880b5c00 msgr2=0x7fa5880b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 shutdown_connections 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.176+0000 7fa58dcff640 1 -- 172.21.15.143:0/2428642352 wait complete. 2024-03-25T19:33:15.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:15.177+0000 7fa58dcff640 1 librados: shutdown 2024-03-25T19:33:15.189 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:21.995 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.111+0000 7f0012993640 10 monclient: get_monmap_and_config 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.111+0000 7f0012993640 10 monclient: build_initial_monmap 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.111+0000 7f0012993640 10 monclient: monmap: 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:33:22.112225+0000 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:33:22.112225+0000 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:33:22.111 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:33:22.112 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 1 Processor -- start 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 1 -- start start 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 10 monclient: init 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 10 monclient(hunting): picked mon.noname-c con 0x7f000c0b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 10 monclient(hunting): picked mon.noname-a con 0x7f000c0b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 10 monclient(hunting): picked mon.noname-b con 0x7f000c14be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.112+0000 7f0012993640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c0af940 con 0x7f000c0b1730 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.113+0000 7f0012993640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c0b0a70 con 0x7f000c0b4d60 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.113+0000 7f0012993640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c0b0d50 con 0x7f000c14be90 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.113+0000 7f0012993640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:22.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.113+0000 7f0012993640 10 monclient(hunting): authenticate will time out at 6218.456671s 2024-03-25T19:33:22.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.122+0000 7f000bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f000c0b4d60 0x7f000c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:55952/0 (socket says 172.21.15.143:55952) 2024-03-25T19:33:22.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.122+0000 7f000bfff640 1 -- 172.21.15.143:0/4103396075 learned_addr learned my addr 172.21.15.143:0/4103396075 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:22.128 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.128+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269414767 0 0) 0x7f000c0b0a70 con 0x7f000c0b4d60 2024-03-25T19:33:22.128 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.128+0000 7f000affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.128 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.128+0000 7f000affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.128 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.128+0000 7f000affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.128 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.128+0000 7f000affd640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:33:22.128 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.128+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff4003520 con 0x7f000c0b4d60 2024-03-25T19:33:22.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.135+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445243958 0 0) 0x7f000c0af940 con 0x7f000c0b1730 2024-03-25T19:33:22.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.135+0000 7f000affd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.135+0000 7f000affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.135+0000 7f000affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.135+0000 7f000affd640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:33:22.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.135+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f000c0b0a70 con 0x7f000c0b1730 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899006901 0 0) 0x7efff4003520 con 0x7f000c0b4d60 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 >> v1:172.21.15.143:6790/0 conn(0x7f000c14be90 legacy=0x7f000c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 >> v1:172.21.15.143:6789/0 conn(0x7f000c0b1730 legacy=0x7f000c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 1 monclient: found mon.noname-c 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c0b1030 con 0x7f000c0b4d60 2024-03-25T19:33:22.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 10 monclient: _finish_auth 0 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efffc001690 con 0x7f000c0b4d60 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f0012993640 5 monclient: authenticate success, global_id 4378 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f0012993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f000affd640 10 monclient: dump: 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:22.143 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:22.144 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:22.144 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:22.144 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:22.144 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:22.144 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:22.144 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:22.144 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:22.144 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.142+0000 7f0012993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:22.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.145+0000 7f000affd640 1 -- 172.21.15.143:0/4103396075 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efffc0017f0 con 0x7f000c0b4d60 2024-03-25T19:33:22.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.145+0000 7f000affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:22.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.145+0000 7f0012993640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:22.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.145+0000 7f0012993640 4 set_mon_vals no callback set 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.146+0000 7f0012993640 10 monclient: shutdown 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.146+0000 7f0012993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.146+0000 7f0012993640 1 -- 172.21.15.143:0/4103396075 >> v1:172.21.15.203:6789/0 conn(0x7f000c0b4d60 legacy=0x7f000c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.146+0000 7f0012993640 1 -- 172.21.15.143:0/4103396075 shutdown_connections 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.146+0000 7f0012993640 1 -- 172.21.15.143:0/4103396075 >> 172.21.15.143:0/4103396075 conn(0x7f000c0617a0 msgr2=0x7f000c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.146+0000 7f0012993640 1 -- 172.21.15.143:0/4103396075 shutdown_connections 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.146+0000 7f0012993640 1 -- 172.21.15.143:0/4103396075 wait complete. 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.147+0000 7f0012993640 10 monclient: build_initial_monmap 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.147+0000 7f0012993640 10 monclient: monmap: 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:22.147 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.147+0000 7f0012993640 1 librados: starting msgr at 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.147+0000 7f0012993640 1 librados: starting objecter 2024-03-25T19:33:22.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.147+0000 7f0012993640 1 Processor -- start 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 1 -- start start 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 1 librados: setting wanted keys 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 1 librados: calling monclient init 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 10 monclient: init 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 10 monclient(hunting): picked mon.noname-c con 0x7f000c0b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 10 monclient(hunting): picked mon.noname-b con 0x7f000c0b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.148+0000 7f0012993640 10 monclient(hunting): picked mon.noname-a con 0x7f000c14be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.149+0000 7f0012993640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c135810 con 0x7f000c14be90 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.149+0000 7f0012993640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c13a630 con 0x7f000c0b4d60 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.149+0000 7f0012993640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c1dac70 con 0x7f000c0b1730 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.149+0000 7f0012993640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:22.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.149+0000 7f0012993640 10 monclient(hunting): authenticate will time out at 6218.492851s 2024-03-25T19:33:22.155 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.155+0000 7f000b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f000c0b4d60 0x7f000c1349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:55988/0 (socket says 172.21.15.143:55988) 2024-03-25T19:33:22.155 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.155+0000 7f000b7fe640 1 -- 172.21.15.143:0/2313685496 learned_addr learned my addr 172.21.15.143:0/2313685496 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:22.157 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.157+0000 7f0008ff9640 10 client.?.objecter ms_handle_connect 0x7f000c0b4d60 2024-03-25T19:33:22.157 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.157+0000 7f0008ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:22.161 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.162+0000 7f0008ff9640 10 client.?.objecter ms_handle_connect 0x7f000c0b1730 2024-03-25T19:33:22.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.162+0000 7f0008ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:22.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.165+0000 7f0008ff9640 10 client.?.objecter ms_handle_connect 0x7f000c14be90 2024-03-25T19:33:22.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.165+0000 7f0008ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:22.166 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.167+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932837794 0 0) 0x7f000c13a630 con 0x7f000c0b4d60 2024-03-25T19:33:22.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.167+0000 7f0008ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.167+0000 7f0008ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.167+0000 7f0008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.167+0000 7f0008ff9640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:33:22.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.167+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff8004eb0 con 0x7f000c0b4d60 2024-03-25T19:33:22.176 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.177+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415344 0 0) 0x7f000c135810 con 0x7f000c14be90 2024-03-25T19:33:22.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.177+0000 7f0008ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.177+0000 7f0008ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.177+0000 7f0008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.177+0000 7f0008ff9640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:33:22.177 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.177+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f000c13a630 con 0x7f000c14be90 2024-03-25T19:33:22.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.184+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137688979 0 0) 0x7f000c1dac70 con 0x7f000c0b1730 2024-03-25T19:33:22.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.184+0000 7f0008ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:22.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.184+0000 7f0008ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:22.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.184+0000 7f0008ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:22.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.184+0000 7f0008ff9640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:33:22.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.184+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f000c135810 con 0x7f000c0b1730 2024-03-25T19:33:22.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.185+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2116669063 0 0) 0x7efff8004eb0 con 0x7f000c0b4d60 2024-03-25T19:33:22.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.185+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f000c1dac70 con 0x7f000c0b4d60 2024-03-25T19:33:22.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.185+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0000001a10 con 0x7f000c0b4d60 2024-03-25T19:33:22.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.185+0000 7f0008ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:22.185 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.185+0000 7f0008ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.185+0000 7f0008ff9640 10 monclient(hunting): dump: 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:22.186 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.186+0000 7f0008ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:22.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.195+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2375658552 0 0) 0x7f000c135810 con 0x7f000c0b1730 2024-03-25T19:33:22.194 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.195+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efff8004eb0 con 0x7f000c0b1730 2024-03-25T19:33:22.196 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1094378677 0 0) 0x7f000c1dac70 con 0x7f000c0b4d60 2024-03-25T19:33:22.196 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 >> v1:172.21.15.143:6790/0 conn(0x7f000c0b1730 legacy=0x7f000c1db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.196 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 >> v1:172.21.15.143:6789/0 conn(0x7f000c14be90 legacy=0x7f000c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.196 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 1 monclient: found mon.b 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c1428f0 con 0x7f000c0b4d60 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0008ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 10 monclient: _renew_subs 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000c143220 con 0x7f000c0b4d60 2024-03-25T19:33:22.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:22.198 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 10 monclient: _renew_subs 2024-03-25T19:33:22.198 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:22.198 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c1dba80 con 0x7f000c0b4d60 2024-03-25T19:33:22.198 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.196+0000 7f0012993640 1 librados: init done 2024-03-25T19:33:22.198 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.198+0000 7f0012993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:22.198 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.198+0000 7f0012993640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:22.199 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.198+0000 7f0012993640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:22.199 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.198+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c143460 con 0x7f000c0b4d60 2024-03-25T19:33:22.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.479+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0000001c50 con 0x7f000c0b4d60 2024-03-25T19:33:22.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.479+0000 7f0008ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:22.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.479+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f00000038c0 con 0x7f000c0b4d60 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.479+0000 7f0008ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.479+0000 7f0008ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.479+0000 7f0008ff9640 10 monclient: dump: 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:22.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.479+0000 7f000a7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:22.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.480+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f0000003ae0 con 0x7f000c0b4d60 2024-03-25T19:33:22.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.481+0000 7f000bfff640 10 monclient: get_auth_request con 0x7efff806e070 auth_method 0 2024-03-25T19:33:22.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.481+0000 7f0008ff9640 10 client.4381.objecter ms_handle_connect 0x7efff806e070 2024-03-25T19:33:22.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.485+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f00000546f0 con 0x7f000c0b4d60 2024-03-25T19:33:22.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.485+0000 7f0008ff9640 10 client.4381.objecter ms_dispatch 0x7f000c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:22.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.485+0000 7f0008ff9640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:22.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.485+0000 7f0008ff9640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:22.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.485+0000 7f0008ff9640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T19:33:22.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.486+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0000050200 con 0x7f000c0b4d60 2024-03-25T19:33:22.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.486+0000 7f0008ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:22.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.486+0000 7f0008ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:22.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.727+0000 7f0012993640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:22.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.727+0000 7f0012993640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:22.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.727+0000 7f0012993640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:22.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.727+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f000c0627b0 con 0x7f000c0b4d60 2024-03-25T19:33:22.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.752+0000 7f0008ff9640 1 -- 172.21.15.143:0/2313685496 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f0000052130 con 0x7f000c0b4d60 2024-03-25T19:33:22.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.752+0000 7f0008ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:22.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.752+0000 7f0008ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:22.752 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:33:22.753 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 10 librados: watch_flush enter 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 10 librados: watch_flush exit 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 10 client.4381.objecter successfully canceled tick 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 >> v1:172.21.15.143:6800/1292282666 conn(0x7efff806e070 legacy=0x7efff8070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 10 monclient: shutdown 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 >> v1:172.21.15.203:6789/0 conn(0x7f000c0b4d60 legacy=0x7f000c1349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 shutdown_connections 2024-03-25T19:33:22.756 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.755+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 >> 172.21.15.143:0/2313685496 conn(0x7f000c0617a0 msgr2=0x7f000c137400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:22.757 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.756+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 shutdown_connections 2024-03-25T19:33:22.757 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.756+0000 7f0012993640 1 -- 172.21.15.143:0/2313685496 wait complete. 2024-03-25T19:33:22.757 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:22.756+0000 7f0012993640 1 librados: shutdown 2024-03-25T19:33:22.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:29.574 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:29.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.692+0000 7f5d33b97640 10 monclient: get_monmap_and_config 2024-03-25T19:33:29.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.692+0000 7f5d33b97640 10 monclient: build_initial_monmap 2024-03-25T19:33:29.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.692+0000 7f5d33b97640 10 monclient: monmap: 2024-03-25T19:33:29.693 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:29.693 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:29.693 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:33:29.694618+0000 2024-03-25T19:33:29.694 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:33:29.694618+0000 2024-03-25T19:33:29.694 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:29.694 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:29.694 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:29.694 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:33:29.694 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:33:29.694 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:29.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.693+0000 7f5d33b97640 1 Processor -- start 2024-03-25T19:33:29.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.693+0000 7f5d33b97640 1 -- start start 2024-03-25T19:33:29.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.693+0000 7f5d33b97640 10 monclient: init 2024-03-25T19:33:29.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:29.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:33:29.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 10 monclient(hunting): picked mon.noname-b con 0x7f5d2c0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:29.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2c13f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:29.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 10 monclient(hunting): picked mon.noname-a con 0x7f5d2c1430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:29.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c142520 con 0x7f5d2c1430c0 2024-03-25T19:33:29.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c0b28a0 con 0x7f5d2c13f710 2024-03-25T19:33:29.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c0b2a30 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:29.698 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.694+0000 7f5d33b97640 10 monclient(hunting): authenticate will time out at 6226.039560s 2024-03-25T19:33:29.703 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.703+0000 7f5d31132640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5d2c13f710 0x7f5d2c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:56516/0 (socket says 172.21.15.143:56516) 2024-03-25T19:33:29.703 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.703+0000 7f5d31132640 1 -- 172.21.15.143:0/2272287432 learned_addr learned my addr 172.21.15.143:0/2272287432 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:29.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.716+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426195987 0 0) 0x7f5d2c0b28a0 con 0x7f5d2c13f710 2024-03-25T19:33:29.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.716+0000 7f5d30931640 10 monclient(hunting): _negotiate 2024-03-25T19:33:29.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.716+0000 7f5d30931640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:29.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.716+0000 7f5d30931640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:29.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.716+0000 7f5d30931640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:33:29.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.716+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d18003520 con 0x7f5d2c13f710 2024-03-25T19:33:29.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.719+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116800714 0 0) 0x7f5d2c0b2a30 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.719+0000 7f5d30931640 10 monclient(hunting): _negotiate 2024-03-25T19:33:29.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.719+0000 7f5d30931640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:29.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.719+0000 7f5d30931640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:29.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.719+0000 7f5d30931640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:33:29.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.719+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c0b28a0 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3686787348 0 0) 0x7f5d2c0b28a0 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 >> v1:172.21.15.203:6789/0 conn(0x7f5d2c13f710 legacy=0x7f5d2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:29.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 >> v1:172.21.15.143:6789/0 conn(0x7f5d2c1430c0 legacy=0x7f5d2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:29.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 10 monclient: _finish_hunting 0 2024-03-25T19:33:29.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 1 monclient: found mon.noname-b 2024-03-25T19:33:29.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:29.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c0b44c0 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 10 monclient: _finish_auth 0 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 10 monclient: _check_auth_tickets 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5d240019f0 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d33b97640 5 monclient: authenticate success, global_id 4370 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d33b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d30931640 10 monclient: dump: 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:29.731 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:29.732 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:29.732 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:29.732 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:29.732 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:29.732 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:29.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.729+0000 7f5d33b97640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:29.740 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.740+0000 7f5d30931640 1 -- 172.21.15.143:0/2272287432 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d24001b50 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.740+0000 7f5d30931640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:29.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.740+0000 7f5d33b97640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:29.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.740+0000 7f5d33b97640 4 set_mon_vals no callback set 2024-03-25T19:33:29.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.740+0000 7f5d33b97640 10 monclient: shutdown 2024-03-25T19:33:29.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.740+0000 7f5d33b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:29.742 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.740+0000 7f5d33b97640 1 -- 172.21.15.143:0/2272287432 >> v1:172.21.15.143:6790/0 conn(0x7f5d2c0a98f0 legacy=0x7f5d2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:29.742 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.741+0000 7f5d33b97640 1 -- 172.21.15.143:0/2272287432 shutdown_connections 2024-03-25T19:33:29.742 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.741+0000 7f5d33b97640 1 -- 172.21.15.143:0/2272287432 >> 172.21.15.143:0/2272287432 conn(0x7f5d2c0b7ea0 msgr2=0x7f5d2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:29.742 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.741+0000 7f5d33b97640 1 -- 172.21.15.143:0/2272287432 shutdown_connections 2024-03-25T19:33:29.742 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.741+0000 7f5d33b97640 1 -- 172.21.15.143:0/2272287432 wait complete. 2024-03-25T19:33:29.742 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 10 monclient: build_initial_monmap 2024-03-25T19:33:29.742 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 10 monclient: monmap: 2024-03-25T19:33:29.742 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 1 librados: starting msgr at 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 1 librados: starting objecter 2024-03-25T19:33:29.743 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 1 Processor -- start 2024-03-25T19:33:29.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 1 -- start start 2024-03-25T19:33:29.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 1 librados: setting wanted keys 2024-03-25T19:33:29.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 1 librados: calling monclient init 2024-03-25T19:33:29.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.742+0000 7f5d33b97640 10 monclient: init 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 10 monclient(hunting): picked mon.noname-b con 0x7f5d2c0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 10 monclient(hunting): picked mon.noname-a con 0x7f5d2c13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2c1430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c070040 con 0x7f5d2c13f710 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c071fc0 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c06eeb0 con 0x7f5d2c1430c0 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:29.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.743+0000 7f5d33b97640 10 monclient(hunting): authenticate will time out at 6226.087980s 2024-03-25T19:33:29.755 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.754+0000 7f5d31132640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5d2c13f710 0x7f5d2c070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:41900/0 (socket says 172.21.15.143:41900) 2024-03-25T19:33:29.755 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.754+0000 7f5d31132640 1 -- 172.21.15.143:0/3955675245 learned_addr learned my addr 172.21.15.143:0/3955675245 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:29.757 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.756+0000 7f5d227fc640 10 client.?.objecter ms_handle_connect 0x7f5d2c13f710 2024-03-25T19:33:29.757 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.756+0000 7f5d227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:29.759 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.759+0000 7f5d227fc640 10 client.?.objecter ms_handle_connect 0x7f5d2c1430c0 2024-03-25T19:33:29.760 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.759+0000 7f5d227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:29.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.760+0000 7f5d227fc640 10 client.?.objecter ms_handle_connect 0x7f5d2c0a98f0 2024-03-25T19:33:29.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.760+0000 7f5d227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:29.771 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.770+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672217550 0 0) 0x7f5d2c06eeb0 con 0x7f5d2c1430c0 2024-03-25T19:33:29.771 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.770+0000 7f5d227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:29.771 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.770+0000 7f5d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:29.771 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.770+0000 7f5d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:29.771 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.770+0000 7f5d227fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:33:29.771 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.770+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d10004eb0 con 0x7f5d2c1430c0 2024-03-25T19:33:29.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.772+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084255802 0 0) 0x7f5d2c070040 con 0x7f5d2c13f710 2024-03-25T19:33:29.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.772+0000 7f5d227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:29.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.772+0000 7f5d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:29.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.772+0000 7f5d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:29.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.772+0000 7f5d227fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:33:29.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.772+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c06eeb0 con 0x7f5d2c13f710 2024-03-25T19:33:29.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.776+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688606453 0 0) 0x7f5d2c071fc0 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.776+0000 7f5d227fc640 10 monclient(hunting): _negotiate 2024-03-25T19:33:29.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.776+0000 7f5d227fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:29.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.776+0000 7f5d227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:29.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.776+0000 7f5d227fc640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:33:29.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.776+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c070040 con 0x7f5d2c0a98f0 2024-03-25T19:33:29.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.782+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 636852221 0 0) 0x7f5d2c06eeb0 con 0x7f5d2c13f710 2024-03-25T19:33:29.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.782+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d2c071fc0 con 0x7f5d2c13f710 2024-03-25T19:33:29.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.782+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5d1c0018d0 con 0x7f5d2c13f710 2024-03-25T19:33:29.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.782+0000 7f5d227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.782+0000 7f5d227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.782+0000 7f5d227fc640 10 monclient(hunting): dump: 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:29.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.782+0000 7f5d227fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 616221543 0 0) 0x7f5d2c071fc0 con 0x7f5d2c13f710 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 >> v1:172.21.15.143:6790/0 conn(0x7f5d2c1430c0 legacy=0x7f5d2c1e5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 >> v1:172.21.15.203:6789/0 conn(0x7f5d2c0a98f0 legacy=0x7f5d2c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 10 monclient: _finish_hunting 0 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 1 monclient: found mon.a 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c06f040 con 0x7f5d2c13f710 2024-03-25T19:33:29.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 10 monclient: _finish_auth 0 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 10 monclient: _check_auth_tickets 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 5 monclient: authenticate success, global_id 4383 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 10 monclient: _renew_subs 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c14bf00 con 0x7f5d2c13f710 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 10 monclient: _renew_subs 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c1395e0 con 0x7f5d2c13f710 2024-03-25T19:33:29.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.794+0000 7f5d33b97640 1 librados: init done 2024-03-25T19:33:29.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.795+0000 7f5d33b97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:29.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.795+0000 7f5d33b97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:29.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.795+0000 7f5d33b97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:29.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:29.795+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c14c090 con 0x7f5d2c13f710 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.110+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d1c001d00 con 0x7f5d2c13f710 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.110+0000 7f5d227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.110+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5d1c004820 con 0x7f5d2c13f710 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.110+0000 7f5d227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.110+0000 7f5d227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.110+0000 7f5d227fc640 10 monclient: dump: 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:30.111 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:30.112 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:30.112 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:30.112 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:30.112 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:30.112 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:30.112 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:30.112 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:30.112 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.110+0000 7f5d23fff640 4 set_mon_vals no callback set 2024-03-25T19:33:30.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.111+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f5d1c004a40 con 0x7f5d2c13f710 2024-03-25T19:33:30.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.112+0000 7f5d31933640 10 monclient: get_auth_request con 0x7f5d1006de20 auth_method 0 2024-03-25T19:33:30.113 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.112+0000 7f5d227fc640 10 client.4383.objecter ms_handle_connect 0x7f5d1006de20 2024-03-25T19:33:30.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.119+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f5d1c054640 con 0x7f5d2c13f710 2024-03-25T19:33:30.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.119+0000 7f5d227fc640 10 client.4383.objecter ms_dispatch 0x7f5d2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:30.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.119+0000 7f5d227fc640 3 client.4383.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:30.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.119+0000 7f5d227fc640 3 client.4383.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:30.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.119+0000 7f5d227fc640 20 client.4383.objecter dump_active .. 0 homeless 2024-03-25T19:33:30.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.119+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d1c04ff80 con 0x7f5d2c13f710 2024-03-25T19:33:30.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.119+0000 7f5d227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:30.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.119+0000 7f5d227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:30.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.369+0000 7f5d33b97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:30.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.369+0000 7f5d33b97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:30.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.369+0000 7f5d33b97640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:30.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.369+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d2c06fe40 con 0x7f5d2c13f710 2024-03-25T19:33:30.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.393+0000 7f5d227fc640 1 -- 172.21.15.143:0/3955675245 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f5d1c052080 con 0x7f5d2c13f710 2024-03-25T19:33:30.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.393+0000 7f5d227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:30.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.393+0000 7f5d227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:30.393 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:33:30.393 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:30.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 10 librados: watch_flush enter 2024-03-25T19:33:30.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 10 librados: watch_flush exit 2024-03-25T19:33:30.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 20 client.4383.objecter shutdown clearing up homeless session... 2024-03-25T19:33:30.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 10 client.4383.objecter successfully canceled tick 2024-03-25T19:33:30.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 >> v1:172.21.15.143:6800/1292282666 conn(0x7f5d1006de20 legacy=0x7f5d100702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 10 monclient: shutdown 2024-03-25T19:33:30.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:30.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 >> v1:172.21.15.143:6789/0 conn(0x7f5d2c13f710 legacy=0x7f5d2c070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:30.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 shutdown_connections 2024-03-25T19:33:30.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 >> 172.21.15.143:0/3955675245 conn(0x7f5d2c0b7ea0 msgr2=0x7f5d2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:30.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 shutdown_connections 2024-03-25T19:33:30.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 1 -- 172.21.15.143:0/3955675245 wait complete. 2024-03-25T19:33:30.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:30.395+0000 7f5d33b97640 1 librados: shutdown 2024-03-25T19:33:30.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:33.315 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:59.880 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:33.320 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:59.885 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:33.393 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:03:59.958 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:33.431 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:59.966 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:33.435 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:04:00.000 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:33.437 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:03:59.973 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:33.440 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:04:00.005 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:33.475 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:04:00.039 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:33.475 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:00.010 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:33:37.211 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:37.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.325+0000 7ff2e8bcf640 10 monclient: get_monmap_and_config 2024-03-25T19:33:37.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.325+0000 7ff2e8bcf640 10 monclient: build_initial_monmap 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.325+0000 7ff2e8bcf640 10 monclient: monmap: 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:33:37.326081+0000 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:33:37.326081+0000 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:33:37.325 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:37.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.325+0000 7ff2e8bcf640 1 Processor -- start 2024-03-25T19:33:37.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 1 -- start start 2024-03-25T19:33:37.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 10 monclient: init 2024-03-25T19:33:37.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:37.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:37.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 10 monclient(hunting): picked mon.noname-c con 0x7ff2e40b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 10 monclient(hunting): picked mon.noname-a con 0x7ff2e40b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 10 monclient(hunting): picked mon.noname-b con 0x7ff2e414be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e40af940 con 0x7ff2e40b1730 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e40b0a70 con 0x7ff2e40b4d60 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e40b0d50 con 0x7ff2e414be90 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.326+0000 7ff2e8bcf640 10 monclient(hunting): authenticate will time out at 6233.670369s 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.328+0000 7ff2e2d9d640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7ff2e414be90 0x7ff2e414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:51582/0 (socket says 172.21.15.143:51582) 2024-03-25T19:33:37.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.328+0000 7ff2e2d9d640 1 -- 172.21.15.143:0/3167378999 learned_addr learned my addr 172.21.15.143:0/3167378999 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:37.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.336+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279037331 0 0) 0x7ff2e40b0d50 con 0x7ff2e414be90 2024-03-25T19:33:37.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.336+0000 7ff2e1d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.336+0000 7ff2e1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.336+0000 7ff2e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.336+0000 7ff2e1d9b640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:33:37.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.336+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d0003520 con 0x7ff2e414be90 2024-03-25T19:33:37.341 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 937432637 0 0) 0x7ff2d0003520 con 0x7ff2e414be90 2024-03-25T19:33:37.341 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 >> v1:172.21.15.203:6789/0 conn(0x7ff2e40b4d60 legacy=0x7ff2e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:37.341 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 >> v1:172.21.15.143:6789/0 conn(0x7ff2e40b1730 legacy=0x7ff2e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:33:37.341 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:33:37.341 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 1 monclient: found mon.noname-b 2024-03-25T19:33:37.341 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:37.341 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:33:37.341 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40b1030 con 0x7ff2e414be90 2024-03-25T19:33:37.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 10 monclient: _finish_auth 0 2024-03-25T19:33:37.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:33:37.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:37.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e8bcf640 5 monclient: authenticate success, global_id 4376 2024-03-25T19:33:37.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.341+0000 7ff2e8bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:37.350 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff2dc001a10 con 0x7ff2e414be90 2024-03-25T19:33:37.350 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.350 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:37.350 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 10 monclient: dump: 2024-03-25T19:33:37.350 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:37.350 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e8bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2dc001ce0 con 0x7ff2e414be90 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e8bcf640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 1 -- 172.21.15.143:0/3167378999 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff2dc002070 con 0x7ff2e414be90 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e8bcf640 4 set_mon_vals no callback set 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.350+0000 7ff2e1d9b640 10 monclient: dump: 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:37.352 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:37.354 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.351+0000 7ff2e8bcf640 10 monclient: shutdown 2024-03-25T19:33:37.354 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.351+0000 7ff2e8bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:37.354 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.351+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/3167378999 >> v1:172.21.15.143:6790/0 conn(0x7ff2e414be90 legacy=0x7ff2e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:37.354 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.351+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/3167378999 shutdown_connections 2024-03-25T19:33:37.354 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.351+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/3167378999 >> 172.21.15.143:0/3167378999 conn(0x7ff2e40617a0 msgr2=0x7ff2e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.351+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/3167378999 shutdown_connections 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.351+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/3167378999 wait complete. 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.352+0000 7ff2e8bcf640 10 monclient: build_initial_monmap 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.352+0000 7ff2e8bcf640 10 monclient: monmap: 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:37.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.352+0000 7ff2e8bcf640 1 librados: starting msgr at 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.352+0000 7ff2e8bcf640 1 librados: starting objecter 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 1 Processor -- start 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 1 -- start start 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 1 librados: setting wanted keys 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 1 librados: calling monclient init 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 10 monclient: init 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 10 monclient(hunting): picked mon.noname-a con 0x7ff2e40b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 10 monclient(hunting): picked mon.noname-b con 0x7ff2e40b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 10 monclient(hunting): picked mon.noname-c con 0x7ff2e414be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e413b340 con 0x7ff2e40b1730 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e413a250 con 0x7ff2e40b4d60 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e413b570 con 0x7ff2e414be90 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:37.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.353+0000 7ff2e8bcf640 10 monclient(hunting): authenticate will time out at 6233.697274s 2024-03-25T19:33:37.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.368+0000 7ff2dbfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ff2e40b4d60 0x7ff2e4135e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:49828/0 (socket says 172.21.15.143:49828) 2024-03-25T19:33:37.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.368+0000 7ff2dbfff640 1 -- 172.21.15.143:0/66047015 learned_addr learned my addr 172.21.15.143:0/66047015 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:37.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.368+0000 7ff2daffd640 10 client.?.objecter ms_handle_connect 0x7ff2e40b4d60 2024-03-25T19:33:37.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.368+0000 7ff2daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:37.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.379+0000 7ff2daffd640 10 client.?.objecter ms_handle_connect 0x7ff2e40b1730 2024-03-25T19:33:37.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.379+0000 7ff2daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:37.382 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.382+0000 7ff2daffd640 10 client.?.objecter ms_handle_connect 0x7ff2e414be90 2024-03-25T19:33:37.382 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.382+0000 7ff2daffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:37.382 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.383+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965272283 0 0) 0x7ff2e413a250 con 0x7ff2e40b4d60 2024-03-25T19:33:37.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.383+0000 7ff2daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.383+0000 7ff2daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.383+0000 7ff2daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.383+0000 7ff2daffd640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:33:37.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.383+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c4004eb0 con 0x7ff2e40b4d60 2024-03-25T19:33:37.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.388+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932671858 0 0) 0x7ff2e413b340 con 0x7ff2e40b1730 2024-03-25T19:33:37.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.388+0000 7ff2daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.388+0000 7ff2daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.388+0000 7ff2daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.388+0000 7ff2daffd640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:33:37.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.388+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e413a250 con 0x7ff2e40b1730 2024-03-25T19:33:37.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.402+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406907450 0 0) 0x7ff2e413b570 con 0x7ff2e414be90 2024-03-25T19:33:37.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.402+0000 7ff2daffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:37.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.402+0000 7ff2daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:37.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.402+0000 7ff2daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:37.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.402+0000 7ff2daffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:33:37.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.402+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e413b340 con 0x7ff2e414be90 2024-03-25T19:33:37.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.409+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2603399675 0 0) 0x7ff2e413a250 con 0x7ff2e40b1730 2024-03-25T19:33:37.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.409+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2e413b570 con 0x7ff2e40b1730 2024-03-25T19:33:37.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.409+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff2d4001810 con 0x7ff2e40b1730 2024-03-25T19:33:37.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.409+0000 7ff2daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.409+0000 7ff2daffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:37.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.409+0000 7ff2daffd640 10 monclient(hunting): dump: 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:37.410 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.409+0000 7ff2daffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3218435519 0 0) 0x7ff2c4004eb0 con 0x7ff2e40b4d60 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2e413a250 con 0x7ff2e40b4d60 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff2cc004080 con 0x7ff2e40b4d60 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 10 monclient(hunting): dump: 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.420 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654050249 0 0) 0x7ff2e413b340 con 0x7ff2e414be90 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2c4004eb0 con 0x7ff2e414be90 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff2dc001830 con 0x7ff2e414be90 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.420+0000 7ff2daffd640 10 monclient(hunting): dump: 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.421 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:37.422 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:37.422 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:37.422 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:37.422 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:37.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 323195322 0 0) 0x7ff2e413b570 con 0x7ff2e40b1730 2024-03-25T19:33:37.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 >> v1:172.21.15.143:6790/0 conn(0x7ff2e414be90 legacy=0x7ff2e41daae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:37.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 >> v1:172.21.15.203:6789/0 conn(0x7ff2e40b4d60 legacy=0x7ff2e4135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:37.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:37.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 1 monclient: found mon.a 2024-03-25T19:33:37.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:37.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:37.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4139e90 con 0x7ff2e40b1730 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 5 monclient: authenticate success, global_id 4386 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 10 monclient: _renew_subs 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e41390c0 con 0x7ff2e40b1730 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 10 monclient: _renew_subs 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e4136740 con 0x7ff2e40b1730 2024-03-25T19:33:37.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.428+0000 7ff2e8bcf640 1 librados: init done 2024-03-25T19:33:37.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.429+0000 7ff2e8bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:37.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.429+0000 7ff2e8bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:37.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.429+0000 7ff2e8bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:37.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.429+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e40627b0 con 0x7ff2e40b1730 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.720+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2d4001d00 con 0x7ff2e40b1730 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.720+0000 7ff2daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.720+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff2d4003790 con 0x7ff2e40b1730 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.720+0000 7ff2daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.720+0000 7ff2daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.720+0000 7ff2daffd640 10 monclient: dump: 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:37.720 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:37.721 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:37.721 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:37.721 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:37.721 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:37.721 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:37.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.720+0000 7ff2e159a640 4 set_mon_vals no callback set 2024-03-25T19:33:37.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.721+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7ff2d40197a0 con 0x7ff2e40b1730 2024-03-25T19:33:37.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.722+0000 7ff2dbfff640 10 monclient: get_auth_request con 0x7ff2c406e290 auth_method 0 2024-03-25T19:33:37.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.722+0000 7ff2daffd640 10 client.4386.objecter ms_handle_connect 0x7ff2c406e290 2024-03-25T19:33:37.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.728+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7ff2d40534d0 con 0x7ff2e40b1730 2024-03-25T19:33:37.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.728+0000 7ff2daffd640 10 client.4386.objecter ms_dispatch 0x7ff2e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:37.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.728+0000 7ff2daffd640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:37.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.728+0000 7ff2daffd640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:37.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.728+0000 7ff2daffd640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T19:33:37.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.728+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2d404ee10 con 0x7ff2e40b1730 2024-03-25T19:33:37.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.728+0000 7ff2daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:37.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.728+0000 7ff2daffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:37.981 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.981+0000 7ff2e8bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:37.981 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.981+0000 7ff2e8bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:37.981 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.981+0000 7ff2e8bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:33:37.981 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:37.981+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2e413a020 con 0x7ff2e40b1730 2024-03-25T19:33:38.005 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.005+0000 7ff2daffd640 1 -- 172.21.15.143:0/66047015 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7ff2d4050f10 con 0x7ff2e40b1730 2024-03-25T19:33:38.005 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.005+0000 7ff2daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:38.005 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.005+0000 7ff2daffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:38.005 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:33:38.005 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 10 librados: watch_flush enter 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 10 librados: watch_flush exit 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 10 client.4386.objecter successfully canceled tick 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 >> v1:172.21.15.143:6800/1292282666 conn(0x7ff2c406e290 legacy=0x7ff2c4070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 10 monclient: shutdown 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 >> v1:172.21.15.143:6789/0 conn(0x7ff2e40b1730 legacy=0x7ff2e4139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 shutdown_connections 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 >> 172.21.15.143:0/66047015 conn(0x7ff2e40617a0 msgr2=0x7ff2e40b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 shutdown_connections 2024-03-25T19:33:38.008 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 1 -- 172.21.15.143:0/66047015 wait complete. 2024-03-25T19:33:38.009 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:38.008+0000 7ff2e8bcf640 1 librados: shutdown 2024-03-25T19:33:38.021 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:44.827 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:44.938 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.938+0000 7fa6e677a640 10 monclient: get_monmap_and_config 2024-03-25T19:33:44.938 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.938+0000 7fa6e677a640 10 monclient: build_initial_monmap 2024-03-25T19:33:44.938 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.939+0000 7fa6e677a640 10 monclient: monmap: 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:33:44.939916+0000 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:33:44.939916+0000 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:33:44.939 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:44.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.939+0000 7fa6e677a640 1 Processor -- start 2024-03-25T19:33:44.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.939+0000 7fa6e677a640 1 -- start start 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.939+0000 7fa6e677a640 10 monclient: init 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 10 monclient(hunting): picked mon.noname-c con 0x7fa6e00b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 10 monclient(hunting): picked mon.noname-a con 0x7fa6e00b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 10 monclient(hunting): picked mon.noname-b con 0x7fa6e014be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e00af940 con 0x7fa6e00b1730 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e00b0a70 con 0x7fa6e00b4d60 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e00b0d50 con 0x7fa6e014be90 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:44.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.940+0000 7fa6e677a640 10 monclient(hunting): authenticate will time out at 6241.284591s 2024-03-25T19:33:44.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.944+0000 7fa6df7fe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fa6e00b1730 0x7fa6e00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:41376/0 (socket says 172.21.15.143:41376) 2024-03-25T19:33:44.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.944+0000 7fa6df7fe640 1 -- 172.21.15.143:0/4179490410 learned_addr learned my addr 172.21.15.143:0/4179490410 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:44.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.951+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763843185 0 0) 0x7fa6e00af940 con 0x7fa6e00b1730 2024-03-25T19:33:44.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.951+0000 7fa6deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:44.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.951+0000 7fa6deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:44.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.951+0000 7fa6deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:44.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.951+0000 7fa6deffd640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:33:44.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.951+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c8003520 con 0x7fa6e00b1730 2024-03-25T19:33:44.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.958+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070658119 0 0) 0x7fa6e00b0d50 con 0x7fa6e014be90 2024-03-25T19:33:44.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.958+0000 7fa6deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:44.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.958+0000 7fa6deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:44.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.958+0000 7fa6deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:44.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.958+0000 7fa6deffd640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:33:44.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.958+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e00af940 con 0x7fa6e014be90 2024-03-25T19:33:44.964 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768251172 0 0) 0x7fa6e00b0a70 con 0x7fa6e00b4d60 2024-03-25T19:33:44.964 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:33:44.964 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e00b0d50 con 0x7fa6e00b4d60 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1635499584 0 0) 0x7fa6c8003520 con 0x7fa6e00b1730 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 >> v1:172.21.15.143:6790/0 conn(0x7fa6e014be90 legacy=0x7fa6e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 >> v1:172.21.15.203:6789/0 conn(0x7fa6e00b4d60 legacy=0x7fa6e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient: _finish_hunting 0 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 1 monclient: found mon.noname-a 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e00b1030 con 0x7fa6e00b1730 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient: _finish_auth 0 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient: _check_auth_tickets 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:44.965 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6d4001690 con 0x7fa6e00b1730 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6e677a640 5 monclient: authenticate success, global_id 4389 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6e677a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6deffd640 10 monclient: dump: 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:44.966 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:44.967 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.964+0000 7fa6e677a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:44.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.974+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6d4001870 con 0x7fa6e00b1730 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.974+0000 7fa6deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.974+0000 7fa6e677a640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.974+0000 7fa6e677a640 4 set_mon_vals no callback set 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.974+0000 7fa6deffd640 1 -- 172.21.15.143:0/4179490410 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6d4001c80 con 0x7fa6e00b1730 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.974+0000 7fa6deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.974+0000 7fa6deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.974+0000 7fa6deffd640 10 monclient: dump: 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:44.975 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.975+0000 7fa6e677a640 10 monclient: shutdown 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.975+0000 7fa6e677a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.975+0000 7fa6e677a640 1 -- 172.21.15.143:0/4179490410 >> v1:172.21.15.143:6789/0 conn(0x7fa6e00b1730 legacy=0x7fa6e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.975+0000 7fa6e677a640 1 -- 172.21.15.143:0/4179490410 shutdown_connections 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.975+0000 7fa6e677a640 1 -- 172.21.15.143:0/4179490410 >> 172.21.15.143:0/4179490410 conn(0x7fa6e00617a0 msgr2=0x7fa6e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.975+0000 7fa6e677a640 1 -- 172.21.15.143:0/4179490410 shutdown_connections 2024-03-25T19:33:44.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.975+0000 7fa6e677a640 1 -- 172.21.15.143:0/4179490410 wait complete. 2024-03-25T19:33:44.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 10 monclient: build_initial_monmap 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 10 monclient: monmap: 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 1 librados: starting msgr at 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 1 librados: starting objecter 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 1 Processor -- start 2024-03-25T19:33:44.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 1 -- start start 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 1 librados: setting wanted keys 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 1 librados: calling monclient init 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.976+0000 7fa6e677a640 10 monclient: init 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 10 monclient(hunting): picked mon.noname-a con 0x7fa6e00b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 10 monclient(hunting): picked mon.noname-b con 0x7fa6e00b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 10 monclient(hunting): picked mon.noname-c con 0x7fa6e014be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e0138040 con 0x7fa6e00b1730 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e01422d0 con 0x7fa6e00b4d60 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6e0141ff0 con 0x7fa6e014be90 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:44.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.977+0000 7fa6e677a640 10 monclient(hunting): authenticate will time out at 6241.320851s 2024-03-25T19:33:44.981 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.981+0000 7fa6df7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa6e00b4d60 0x7fa6e0133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:54818/0 (socket says 172.21.15.143:54818) 2024-03-25T19:33:44.981 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.981+0000 7fa6df7fe640 1 -- 172.21.15.143:0/2846424271 learned_addr learned my addr 172.21.15.143:0/2846424271 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:44.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.984+0000 7fa6dcff9640 10 client.?.objecter ms_handle_connect 0x7fa6e00b4d60 2024-03-25T19:33:44.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.984+0000 7fa6dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:44.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.991+0000 7fa6dcff9640 10 client.?.objecter ms_handle_connect 0x7fa6e00b1730 2024-03-25T19:33:44.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.991+0000 7fa6dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:44.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.991+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712398753 0 0) 0x7fa6e01422d0 con 0x7fa6e00b4d60 2024-03-25T19:33:44.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.991+0000 7fa6dcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:44.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.991+0000 7fa6dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:44.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.991+0000 7fa6dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:44.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.991+0000 7fa6dcff9640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:33:44.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.991+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6cc004eb0 con 0x7fa6e00b4d60 2024-03-25T19:33:44.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.995+0000 7fa6dcff9640 10 client.?.objecter ms_handle_connect 0x7fa6e014be90 2024-03-25T19:33:44.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:44.995+0000 7fa6dcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.002+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1513768611 0 0) 0x7fa6cc004eb0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.002+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6e01422d0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.002+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6d4001bc0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.002+0000 7fa6dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.002+0000 7fa6dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:33:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.002+0000 7fa6dcff9640 10 monclient(hunting): dump: 2024-03-25T19:33:45.002 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:45.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.002+0000 7fa6dcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:45.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.013+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758337103 0 0) 0x7fa6e0138040 con 0x7fa6e00b1730 2024-03-25T19:33:45.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.013+0000 7fa6dcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:33:45.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.013+0000 7fa6dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:45.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.013+0000 7fa6dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:45.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.013+0000 7fa6dcff9640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:33:45.013 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.013+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6cc004eb0 con 0x7fa6e00b1730 2024-03-25T19:33:45.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.015+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3050530976 0 0) 0x7fa6e01422d0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 >> v1:172.21.15.143:6790/0 conn(0x7fa6e014be90 legacy=0x7fa6e0134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 >> v1:172.21.15.143:6789/0 conn(0x7fa6e00b1730 legacy=0x7fa6e0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:33:45.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 1 monclient: found mon.b 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e0134a20 con 0x7fa6e00b4d60 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 10 monclient: _finish_auth 0 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 10 monclient: _renew_subs 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e0137bd0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 10 monclient: _renew_subs 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e0141000 con 0x7fa6e00b4d60 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.016+0000 7fa6e677a640 1 librados: init done 2024-03-25T19:33:45.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.017+0000 7fa6e677a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:45.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.017+0000 7fa6e677a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:45.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.017+0000 7fa6e677a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:45.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.017+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6e01401f0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.303+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6d4004bf0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.303+0000 7fa6dcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.303+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa6d4004f00 con 0x7fa6e00b4d60 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.303+0000 7fa6dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.303+0000 7fa6dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.303+0000 7fa6dcff9640 10 monclient: dump: 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:45.303 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:45.304 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:45.304 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:45.304 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:45.304 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:45.304 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:45.304 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:45.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.303+0000 7fa6de7fc640 4 set_mon_vals no callback set 2024-03-25T19:33:45.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.304+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fa6d401aed0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.305+0000 7fa6dffff640 10 monclient: get_auth_request con 0x7fa6cc06dd90 auth_method 0 2024-03-25T19:33:45.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.305+0000 7fa6dcff9640 10 client.4396.objecter ms_handle_connect 0x7fa6cc06dd90 2024-03-25T19:33:45.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.311+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fa6d4054990 con 0x7fa6e00b4d60 2024-03-25T19:33:45.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.311+0000 7fa6dcff9640 10 client.4396.objecter ms_dispatch 0x7fa6e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:45.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.311+0000 7fa6dcff9640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:45.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.311+0000 7fa6dcff9640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:45.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.311+0000 7fa6dcff9640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T19:33:45.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.311+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6d40504a0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.311+0000 7fa6dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:45.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.311+0000 7fa6dcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:45.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.565+0000 7fa6e677a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:45.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.565+0000 7fa6e677a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:45.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.565+0000 7fa6e677a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:33:45.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.565+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6e00627b0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.588+0000 7fa6dcff9640 1 -- 172.21.15.143:0/2846424271 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fa6d40523d0 con 0x7fa6e00b4d60 2024-03-25T19:33:45.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.588+0000 7fa6dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:45.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.588+0000 7fa6dcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:45.588 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:33:45.588 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 10 librados: watch_flush enter 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 10 librados: watch_flush exit 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 10 client.4396.objecter successfully canceled tick 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 >> v1:172.21.15.143:6800/1292282666 conn(0x7fa6cc06dd90 legacy=0x7fa6cc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 10 monclient: shutdown 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 >> v1:172.21.15.203:6789/0 conn(0x7fa6e00b4d60 legacy=0x7fa6e0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 shutdown_connections 2024-03-25T19:33:45.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 >> 172.21.15.143:0/2846424271 conn(0x7fa6e00617a0 msgr2=0x7fa6e013c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:45.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 shutdown_connections 2024-03-25T19:33:45.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 1 -- 172.21.15.143:0/2846424271 wait complete. 2024-03-25T19:33:45.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:45.591+0000 7fa6e677a640 1 librados: shutdown 2024-03-25T19:33:45.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:33:52.409 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:33:52.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.526+0000 7f1cead90640 10 monclient: get_monmap_and_config 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.526+0000 7f1cead90640 10 monclient: build_initial_monmap 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.526+0000 7f1cead90640 10 monclient: monmap: 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:33:52.527848+0000 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:33:52.527848+0000 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:33:52.527 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.527+0000 7f1cead90640 1 Processor -- start 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.527+0000 7f1cead90640 1 -- start start 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 10 monclient: init 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce40a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce413f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce41430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce4142520 con 0x7f1ce40a98f0 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce40b28a0 con 0x7f1ce413f710 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce40b2a30 con 0x7f1ce41430c0 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:52.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.528+0000 7f1cead90640 10 monclient(hunting): authenticate will time out at 6248.872529s 2024-03-25T19:33:52.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.533+0000 7f1ce932d640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f1ce41430c0 0x7f1ce414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:40268/0 (socket says 172.21.15.143:40268) 2024-03-25T19:33:52.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.533+0000 7f1ce932d640 1 -- 172.21.15.143:0/751796084 learned_addr learned my addr 172.21.15.143:0/751796084 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:52.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.543+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608349282 0 0) 0x7f1ce40b28a0 con 0x7f1ce413f710 2024-03-25T19:33:52.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.543+0000 7f1ce37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.543+0000 7f1ce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.543+0000 7f1ce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.543+0000 7f1ce37fe640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:33:52.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.543+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4003520 con 0x7f1ce413f710 2024-03-25T19:33:52.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.547+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445385594 0 0) 0x7f1ce40b2a30 con 0x7f1ce41430c0 2024-03-25T19:33:52.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.547+0000 7f1ce37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.547+0000 7f1ce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.547+0000 7f1ce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.547+0000 7f1ce37fe640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:33:52.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.547+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce40b28a0 con 0x7f1ce41430c0 2024-03-25T19:33:52.556 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 858307743 0 0) 0x7f1cd4003520 con 0x7f1ce413f710 2024-03-25T19:33:52.556 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 >> v1:172.21.15.143:6790/0 conn(0x7f1ce41430c0 legacy=0x7f1ce414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 >> v1:172.21.15.143:6789/0 conn(0x7f1ce40a98f0 legacy=0x7f1ce40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 1 monclient: found mon.noname-c 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce40b44c0 con 0x7f1ce413f710 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 10 monclient: _finish_auth 0 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1cd0001690 con 0x7f1ce413f710 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1cead90640 5 monclient: authenticate success, global_id 4399 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1cead90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:52.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1ce37fe640 10 monclient: dump: 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:52.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.556+0000 7f1cead90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:33:52.577 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.578+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cd00017f0 con 0x7f1ce413f710 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.578+0000 7f1ce37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.578+0000 7f1ce37fe640 1 -- 172.21.15.143:0/751796084 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1cd0001c00 con 0x7f1ce413f710 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.578+0000 7f1cead90640 10 monclient: get_monmap_and_config success 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.578+0000 7f1cead90640 4 set_mon_vals no callback set 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.578+0000 7f1ce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.578+0000 7f1ce37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.578+0000 7f1ce37fe640 10 monclient: dump: 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.578 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:52.579 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:52.579 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:52.579 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:52.579 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:52.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.579+0000 7f1cead90640 10 monclient: shutdown 2024-03-25T19:33:52.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.579+0000 7f1cead90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:52.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.579+0000 7f1cead90640 1 -- 172.21.15.143:0/751796084 >> v1:172.21.15.203:6789/0 conn(0x7f1ce413f710 legacy=0x7f1ce41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.579+0000 7f1cead90640 1 -- 172.21.15.143:0/751796084 shutdown_connections 2024-03-25T19:33:52.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.579+0000 7f1cead90640 1 -- 172.21.15.143:0/751796084 >> 172.21.15.143:0/751796084 conn(0x7f1ce40b7ea0 msgr2=0x7f1ce40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.579+0000 7f1cead90640 1 -- 172.21.15.143:0/751796084 shutdown_connections 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.579+0000 7f1cead90640 1 -- 172.21.15.143:0/751796084 wait complete. 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 10 monclient: build_initial_monmap 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 10 monclient: monmap: 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:52.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 1 librados: starting msgr at 2024-03-25T19:33:52.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 1 librados: starting objecter 2024-03-25T19:33:52.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 1 Processor -- start 2024-03-25T19:33:52.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 1 -- start start 2024-03-25T19:33:52.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 1 librados: setting wanted keys 2024-03-25T19:33:52.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 1 librados: calling monclient init 2024-03-25T19:33:52.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.580+0000 7f1cead90640 10 monclient: init 2024-03-25T19:33:52.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.581+0000 7f1cead90640 10 monclient: _reopen_session rank -1 2024-03-25T19:33:52.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.581+0000 7f1cead90640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:33:52.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.581+0000 7f1cead90640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce40a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:33:52.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.581+0000 7f1cead90640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:33:52.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.582+0000 7f1cead90640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce41430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:33:52.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.582+0000 7f1cead90640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce4139070 con 0x7f1ce41430c0 2024-03-25T19:33:52.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.582+0000 7f1cead90640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce4133710 con 0x7f1ce40a98f0 2024-03-25T19:33:52.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.582+0000 7f1cead90640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce41338a0 con 0x7f1ce413f710 2024-03-25T19:33:52.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.582+0000 7f1cead90640 10 monclient(hunting): _renew_subs 2024-03-25T19:33:52.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.582+0000 7f1cead90640 10 monclient(hunting): authenticate will time out at 6248.925936s 2024-03-25T19:33:52.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.585+0000 7f1ce8b2c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f1ce40a98f0 0x7f1ce41378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:47124/0 (socket says 172.21.15.143:47124) 2024-03-25T19:33:52.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.585+0000 7f1ce8b2c640 1 -- 172.21.15.143:0/3264559429 learned_addr learned my addr 172.21.15.143:0/3264559429 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:33:52.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.588+0000 7f1ce17fa640 10 client.?.objecter ms_handle_connect 0x7f1ce40a98f0 2024-03-25T19:33:52.588 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.588+0000 7f1ce17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.593 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.594+0000 7f1ce17fa640 10 client.?.objecter ms_handle_connect 0x7f1ce413f710 2024-03-25T19:33:52.593 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.594+0000 7f1ce17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.610+0000 7f1ce17fa640 10 client.?.objecter ms_handle_connect 0x7f1ce41430c0 2024-03-25T19:33:52.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.610+0000 7f1ce17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:33:52.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.676+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927534870 0 0) 0x7f1ce4139070 con 0x7f1ce41430c0 2024-03-25T19:33:52.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.676+0000 7f1ce17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.676+0000 7f1ce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.676+0000 7f1ce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.676+0000 7f1ce17fa640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:33:52.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.676+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc8004eb0 con 0x7f1ce41430c0 2024-03-25T19:33:52.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.680+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294157656 0 0) 0x7f1ce41338a0 con 0x7f1ce413f710 2024-03-25T19:33:52.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.680+0000 7f1ce17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.680+0000 7f1ce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.680+0000 7f1ce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.680+0000 7f1ce17fa640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:33:52.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.680+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce4139070 con 0x7f1ce413f710 2024-03-25T19:33:52.689 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.689+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166364910 0 0) 0x7f1ce4133710 con 0x7f1ce40a98f0 2024-03-25T19:33:52.689 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.689+0000 7f1ce17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.689+0000 7f1ce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.689+0000 7f1ce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.689+0000 7f1ce17fa640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.689+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce41338a0 con 0x7f1ce40a98f0 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.690+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1347138403 0 0) 0x7f1ce4139070 con 0x7f1ce413f710 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.690+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce4133710 con 0x7f1ce413f710 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.690+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1cd0001b00 con 0x7f1ce413f710 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.690+0000 7f1ce17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.690+0000 7f1ce17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.690+0000 7f1ce17fa640 10 monclient(hunting): dump: 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:52.690 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:52.691 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:52.691 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:52.691 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:52.691 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:52.691 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:52.691 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:52.691 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:52.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.690+0000 7f1ce17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:33:52.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1495929360 0 0) 0x7f1ce4133710 con 0x7f1ce413f710 2024-03-25T19:33:52.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 >> v1:172.21.15.203:6789/0 conn(0x7f1ce40a98f0 legacy=0x7f1ce41378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 >> v1:172.21.15.143:6789/0 conn(0x7f1ce41430c0 legacy=0x7f1ce4133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 1 monclient: found mon.c 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce4133a30 con 0x7f1ce413f710 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 10 monclient: _finish_auth 0 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1ce17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 5 monclient: authenticate success, global_id 4391 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 10 monclient: _renew_subs 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce40b7100 con 0x7f1ce413f710 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 10 monclient: _renew_subs 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:33:52.700 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce4136e60 con 0x7f1ce413f710 2024-03-25T19:33:52.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.699+0000 7f1cead90640 1 librados: init done 2024-03-25T19:33:52.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.700+0000 7f1cead90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.700+0000 7f1cead90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:52.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.700+0000 7f1cead90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:33:52.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:52.700+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce40627b0 con 0x7f1ce413f710 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.010+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cd0001ca0 con 0x7f1ce413f710 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.010+0000 7f1ce17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.010+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1cd0003a40 con 0x7f1ce413f710 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.010+0000 7f1ce17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.010+0000 7f1ce17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.010+0000 7f1ce17fa640 10 monclient: dump: 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:53.010 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:33:53.011 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:33:53.011 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:33:53.011 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:33:53.011 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:33:53.011 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:33:53.011 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:33:53.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.010+0000 7f1ce2ffd640 4 set_mon_vals no callback set 2024-03-25T19:33:53.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.011+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f1cd0019a10 con 0x7f1ce413f710 2024-03-25T19:33:53.012 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.011+0000 7f1ce8b2c640 10 monclient: get_auth_request con 0x7f1cc806de20 auth_method 0 2024-03-25T19:33:53.012 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.012+0000 7f1ce17fa640 10 client.4391.objecter ms_handle_connect 0x7f1cc806de20 2024-03-25T19:33:53.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.017+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f1cd0053620 con 0x7f1ce413f710 2024-03-25T19:33:53.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.017+0000 7f1ce17fa640 10 client.4391.objecter ms_dispatch 0x7f1ce4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:33:53.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.017+0000 7f1ce17fa640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:33:53.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.017+0000 7f1ce17fa640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:33:53.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.017+0000 7f1ce17fa640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-25T19:33:53.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.017+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1cd004f130 con 0x7f1ce413f710 2024-03-25T19:33:53.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.017+0000 7f1ce17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:33:53.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.017+0000 7f1ce17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:33:53.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.264+0000 7f1cead90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.264+0000 7f1cead90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.264+0000 7f1cead90640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:33:53.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.264+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ce4069f40 con 0x7f1ce413f710 2024-03-25T19:33:53.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.297+0000 7f1ce17fa640 1 -- 172.21.15.143:0/3264559429 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f1cd0051060 con 0x7f1ce413f710 2024-03-25T19:33:53.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.297+0000 7f1ce17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:33:53.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.297+0000 7f1ce17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:33:53.297 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:33:53.297 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 10 librados: watch_flush enter 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 10 librados: watch_flush exit 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 10 client.4391.objecter successfully canceled tick 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 >> v1:172.21.15.143:6800/1292282666 conn(0x7f1cc806de20 legacy=0x7f1cc80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 10 monclient: shutdown 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 >> v1:172.21.15.143:6790/0 conn(0x7f1ce413f710 legacy=0x7f1ce41daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 shutdown_connections 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 >> 172.21.15.143:0/3264559429 conn(0x7f1ce40b7ea0 msgr2=0x7f1ce4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 shutdown_connections 2024-03-25T19:33:53.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 1 -- 172.21.15.143:0/3264559429 wait complete. 2024-03-25T19:33:53.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:33:53.300+0000 7f1cead90640 1 librados: shutdown 2024-03-25T19:33:53.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:00.118 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.236+0000 7f9c6e8ed640 10 monclient: get_monmap_and_config 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.236+0000 7f9c6e8ed640 10 monclient: build_initial_monmap 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.236+0000 7f9c6e8ed640 10 monclient: monmap: 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:34:00.237352+0000 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:34:00.237352+0000 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:00.236 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:34:00.237 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:34:00.237 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.237+0000 7f9c6e8ed640 1 Processor -- start 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.237+0000 7f9c6e8ed640 1 -- start start 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.237+0000 7f9c6e8ed640 10 monclient: init 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 10 monclient(hunting): picked mon.noname-c con 0x7f9c680a96e0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 10 monclient(hunting): picked mon.noname-b con 0x7f9c680b1b10 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 10 monclient(hunting): picked mon.noname-a con 0x7f9c68146060 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:00.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c68140040 con 0x7f9c68146060 2024-03-25T19:34:00.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680b1fa0 con 0x7f9c680a96e0 2024-03-25T19:34:00.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680b24c0 con 0x7f9c680b1b10 2024-03-25T19:34:00.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:00.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.238+0000 7f9c6e8ed640 10 monclient(hunting): authenticate will time out at 6256.582148s 2024-03-25T19:34:00.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.252+0000 7f9c677fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f9c680b1b10 0x7f9c68145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:58220/0 (socket says 172.21.15.143:58220) 2024-03-25T19:34:00.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.252+0000 7f9c677fe640 1 -- 172.21.15.143:0/226246013 learned_addr learned my addr 172.21.15.143:0/226246013 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:00.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.264+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95842053 0 0) 0x7f9c680b24c0 con 0x7f9c680b1b10 2024-03-25T19:34:00.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.264+0000 7f9c66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.264+0000 7f9c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.264+0000 7f9c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.264+0000 7f9c66ffd640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:34:00.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.264+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c50003520 con 0x7f9c680b1b10 2024-03-25T19:34:00.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.269+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947009386 0 0) 0x7f9c680b1fa0 con 0x7f9c680a96e0 2024-03-25T19:34:00.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.269+0000 7f9c66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.269+0000 7f9c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.269+0000 7f9c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.269+0000 7f9c66ffd640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:34:00.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.269+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c680b24c0 con 0x7f9c680a96e0 2024-03-25T19:34:00.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.271+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748995319 0 0) 0x7f9c68140040 con 0x7f9c68146060 2024-03-25T19:34:00.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.271+0000 7f9c66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.271+0000 7f9c66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.271+0000 7f9c66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.271+0000 7f9c66ffd640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:34:00.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.271+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c680b1fa0 con 0x7f9c68146060 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752614478 0 0) 0x7f9c50003520 con 0x7f9c680b1b10 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 >> v1:172.21.15.203:6789/0 conn(0x7f9c680a96e0 legacy=0x7f9c680b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 >> v1:172.21.15.143:6789/0 conn(0x7f9c68146060 legacy=0x7f9c6814dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 1 monclient: found mon.noname-b 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c680b15b0 con 0x7f9c680b1b10 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:00.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9c5c001810 con 0x7f9c680b1b10 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c6e8ed640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c6e8ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c66ffd640 10 monclient: dump: 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:00.280 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:00.281 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:00.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.279+0000 7f9c6e8ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.289+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c5c001970 con 0x7f9c680b1b10 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.289+0000 7f9c66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.289+0000 7f9c66ffd640 1 -- 172.21.15.143:0/226246013 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9c5c001d80 con 0x7f9c680b1b10 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.289+0000 7f9c6e8ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.289+0000 7f9c6e8ed640 4 set_mon_vals no callback set 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.289+0000 7f9c66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.289+0000 7f9c66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.289+0000 7f9c66ffd640 10 monclient: dump: 2024-03-25T19:34:00.289 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:00.290 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:00.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.290+0000 7f9c6e8ed640 10 monclient: shutdown 2024-03-25T19:34:00.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.290+0000 7f9c6e8ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:00.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.290+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/226246013 >> v1:172.21.15.143:6790/0 conn(0x7f9c680b1b10 legacy=0x7f9c68145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.290+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/226246013 shutdown_connections 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.290+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/226246013 >> 172.21.15.143:0/226246013 conn(0x7f9c680b7ea0 msgr2=0x7f9c680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.290+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/226246013 shutdown_connections 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.290+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/226246013 wait complete. 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 10 monclient: build_initial_monmap 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 10 monclient: monmap: 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:34:00.293 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 1 librados: starting msgr at 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 1 librados: starting objecter 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 1 Processor -- start 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 1 -- start start 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 1 librados: setting wanted keys 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 1 librados: calling monclient init 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.291+0000 7f9c6e8ed640 10 monclient: init 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 10 monclient(hunting): picked mon.noname-b con 0x7f9c680a96e0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 10 monclient(hunting): picked mon.noname-c con 0x7f9c68146060 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 10 monclient(hunting): picked mon.noname-a con 0x7f9c680ad670 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c680ae440 con 0x7f9c680ad670 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c68142350 con 0x7f9c680a96e0 2024-03-25T19:34:00.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c681424e0 con 0x7f9c68146060 2024-03-25T19:34:00.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:00.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.292+0000 7f9c6e8ed640 10 monclient(hunting): authenticate will time out at 6256.636417s 2024-03-25T19:34:00.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.297+0000 7f9c67fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9c680a96e0 0x7f9c680adb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:37878/0 (socket says 172.21.15.143:37878) 2024-03-25T19:34:00.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.297+0000 7f9c67fff640 1 -- 172.21.15.143:0/31299024 learned_addr learned my addr 172.21.15.143:0/31299024 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:00.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.298+0000 7f9c64ff9640 10 client.?.objecter ms_handle_connect 0x7f9c680a96e0 2024-03-25T19:34:00.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.298+0000 7f9c64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:00.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.300+0000 7f9c64ff9640 10 client.?.objecter ms_handle_connect 0x7f9c68146060 2024-03-25T19:34:00.300 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.300+0000 7f9c64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:00.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.306+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955849620 0 0) 0x7f9c68142350 con 0x7f9c680a96e0 2024-03-25T19:34:00.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.306+0000 7f9c64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.306+0000 7f9c64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.306+0000 7f9c64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.306+0000 7f9c64ff9640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:34:00.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.306+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c54004eb0 con 0x7f9c680a96e0 2024-03-25T19:34:00.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.313+0000 7f9c64ff9640 10 client.?.objecter ms_handle_connect 0x7f9c680ad670 2024-03-25T19:34:00.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.313+0000 7f9c64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:00.320 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.320+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110645891 0 0) 0x7f9c681424e0 con 0x7f9c68146060 2024-03-25T19:34:00.320 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.320+0000 7f9c64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.320 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.320+0000 7f9c64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.320 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.320+0000 7f9c64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.320 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.320+0000 7f9c64ff9640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:34:00.320 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.320+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c68142350 con 0x7f9c68146060 2024-03-25T19:34:00.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.324+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665259260 0 0) 0x7f9c680ae440 con 0x7f9c680ad670 2024-03-25T19:34:00.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.324+0000 7f9c64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:00.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.324+0000 7f9c64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:00.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.324+0000 7f9c64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:00.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.324+0000 7f9c64ff9640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:34:00.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.324+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c681424e0 con 0x7f9c680ad670 2024-03-25T19:34:00.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.325+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1680131855 0 0) 0x7f9c54004eb0 con 0x7f9c680a96e0 2024-03-25T19:34:00.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.325+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c680ae440 con 0x7f9c680a96e0 2024-03-25T19:34:00.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.325+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9c58001a30 con 0x7f9c680a96e0 2024-03-25T19:34:00.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.325+0000 7f9c64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.325+0000 7f9c64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:00.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.325+0000 7f9c64ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:00.325 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:00.325 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:00.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.325+0000 7f9c64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:00.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.330+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2976240638 0 0) 0x7f9c68142350 con 0x7f9c68146060 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.330+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c54004eb0 con 0x7f9c68146060 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.330+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9c5c001ae0 con 0x7f9c68146060 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.330+0000 7f9c64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.330+0000 7f9c64ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.330+0000 7f9c64ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:00.330 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:00.331 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:00.331 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:00.331 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2564133384 0 0) 0x7f9c680ae440 con 0x7f9c680a96e0 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 >> v1:172.21.15.143:6790/0 conn(0x7f9c68146060 legacy=0x7f9c6813af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 >> v1:172.21.15.143:6789/0 conn(0x7f9c680ad670 legacy=0x7f9c681d7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 1 monclient: found mon.b 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c68142670 con 0x7f9c680a96e0 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 10 monclient: _finish_auth 0 2024-03-25T19:34:00.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 5 monclient: authenticate success, global_id 4408 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 10 monclient: _renew_subs 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6813b690 con 0x7f9c680a96e0 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 10 monclient: _renew_subs 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c681d7b40 con 0x7f9c680a96e0 2024-03-25T19:34:00.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.336+0000 7f9c6e8ed640 1 librados: init done 2024-03-25T19:34:00.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.337+0000 7f9c6e8ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:00.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.337+0000 7f9c6e8ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:00.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.337+0000 7f9c6e8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:00.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.337+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c680ae720 con 0x7f9c680a96e0 2024-03-25T19:34:00.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.615+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c58001c70 con 0x7f9c680a96e0 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.615+0000 7f9c64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.615+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9c58003a60 con 0x7f9c680a96e0 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.615+0000 7f9c64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.615+0000 7f9c64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.615+0000 7f9c64ff9640 10 monclient: dump: 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:00.616 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:00.617 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:00.617 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:00.617 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:00.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.615+0000 7f9c667fc640 4 set_mon_vals no callback set 2024-03-25T19:34:00.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.616+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f9c58003c40 con 0x7f9c680a96e0 2024-03-25T19:34:00.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.616+0000 7f9c677fe640 10 monclient: get_auth_request con 0x7f9c5406df70 auth_method 0 2024-03-25T19:34:00.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.617+0000 7f9c64ff9640 10 client.4408.objecter ms_handle_connect 0x7f9c5406df70 2024-03-25T19:34:00.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.620+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f9c580548d0 con 0x7f9c680a96e0 2024-03-25T19:34:00.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.620+0000 7f9c64ff9640 10 client.4408.objecter ms_dispatch 0x7f9c68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:00.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.620+0000 7f9c64ff9640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:00.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.620+0000 7f9c64ff9640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:00.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.620+0000 7f9c64ff9640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T19:34:00.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.620+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c580503e0 con 0x7f9c680a96e0 2024-03-25T19:34:00.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.620+0000 7f9c64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:00.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.620+0000 7f9c64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:00.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.861+0000 7f9c6e8ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:00.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.861+0000 7f9c6e8ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:00.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.861+0000 7f9c6e8ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:00.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.861+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c680a9ac0 con 0x7f9c680a96e0 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.892+0000 7f9c64ff9640 1 -- 172.21.15.143:0/31299024 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f9c58052310 con 0x7f9c680a96e0 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.892+0000 7f9c64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.892+0000 7f9c64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:34:00.892 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.894+0000 7f9c6e8ed640 10 librados: watch_flush enter 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.894+0000 7f9c6e8ed640 10 librados: watch_flush exit 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.894+0000 7f9c6e8ed640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 10 client.4408.objecter successfully canceled tick 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 >> v1:172.21.15.143:6800/1292282666 conn(0x7f9c5406df70 legacy=0x7f9c54070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 10 monclient: shutdown 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 >> v1:172.21.15.203:6789/0 conn(0x7f9c680a96e0 legacy=0x7f9c680adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 shutdown_connections 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 >> 172.21.15.143:0/31299024 conn(0x7f9c680b7ea0 msgr2=0x7f9c680b6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:00.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 shutdown_connections 2024-03-25T19:34:00.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 1 -- 172.21.15.143:0/31299024 wait complete. 2024-03-25T19:34:00.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:00.895+0000 7f9c6e8ed640 1 librados: shutdown 2024-03-25T19:34:00.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:07.715 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.830+0000 7f8720358640 10 monclient: get_monmap_and_config 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.830+0000 7f8720358640 10 monclient: build_initial_monmap 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.830+0000 7f8720358640 10 monclient: monmap: 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:34:07.832192+0000 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:34:07.832192+0000 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:07.831 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:34:07.832 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:34:07.832 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.831+0000 7f8720358640 1 Processor -- start 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.831+0000 7f8720358640 1 -- start start 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.831+0000 7f8720358640 10 monclient: init 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 10 monclient(hunting): picked mon.noname-c con 0x7f87180a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 10 monclient(hunting): picked mon.noname-b con 0x7f871813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 10 monclient(hunting): picked mon.noname-a con 0x7f87181430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8718142520 con 0x7f87181430c0 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87180b28a0 con 0x7f87180a98f0 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87180b2a30 con 0x7f871813f710 2024-03-25T19:34:07.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:07.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.832+0000 7f8720358640 10 monclient(hunting): authenticate will time out at 6264.177111s 2024-03-25T19:34:07.838 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.837+0000 7f871d8f3640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f871813f710 0x7f87181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:45674/0 (socket says 172.21.15.143:45674) 2024-03-25T19:34:07.838 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.837+0000 7f871d8f3640 1 -- 172.21.15.143:0/1771627505 learned_addr learned my addr 172.21.15.143:0/1771627505 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:07.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.850+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218805536 0 0) 0x7f87180b2a30 con 0x7f871813f710 2024-03-25T19:34:07.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.850+0000 7f871d0f2640 10 monclient(hunting): _negotiate 2024-03-25T19:34:07.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.850+0000 7f871d0f2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:07.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.850+0000 7f871d0f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:07.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.850+0000 7f871d0f2640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:34:07.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.850+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8704003520 con 0x7f871813f710 2024-03-25T19:34:07.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.856+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3827367701 0 0) 0x7f8704003520 con 0x7f871813f710 2024-03-25T19:34:07.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 >> v1:172.21.15.203:6789/0 conn(0x7f87180a98f0 legacy=0x7f87180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:34:07.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 >> v1:172.21.15.143:6789/0 conn(0x7f87181430c0 legacy=0x7f871814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 10 monclient: _finish_hunting 0 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 1 monclient: found mon.noname-b 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87180b44c0 con 0x7f871813f710 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 10 monclient: _finish_auth 0 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 10 monclient: _check_auth_tickets 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f871d0f2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f8720358640 5 monclient: authenticate success, global_id 4400 2024-03-25T19:34:07.858 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.857+0000 7f8720358640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:07.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8708001690 con 0x7f871813f710 2024-03-25T19:34:07.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 10 monclient: dump: 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8708001960 con 0x7f871813f710 2024-03-25T19:34:07.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f8720358640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 1 -- 172.21.15.143:0/1771627505 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8708001cf0 con 0x7f871813f710 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f8720358640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f8720358640 4 set_mon_vals no callback set 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f871d0f2640 10 monclient: dump: 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:07.877 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:07.878 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:07.878 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:07.878 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f8720358640 10 monclient: shutdown 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.875+0000 7f8720358640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.876+0000 7f8720358640 1 -- 172.21.15.143:0/1771627505 >> v1:172.21.15.143:6790/0 conn(0x7f871813f710 legacy=0x7f87181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.876+0000 7f8720358640 1 -- 172.21.15.143:0/1771627505 shutdown_connections 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.876+0000 7f8720358640 1 -- 172.21.15.143:0/1771627505 >> 172.21.15.143:0/1771627505 conn(0x7f87180b7ea0 msgr2=0x7f87180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.876+0000 7f8720358640 1 -- 172.21.15.143:0/1771627505 shutdown_connections 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.876+0000 7f8720358640 1 -- 172.21.15.143:0/1771627505 wait complete. 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.876+0000 7f8720358640 10 monclient: build_initial_monmap 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.876+0000 7f8720358640 10 monclient: monmap: 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:34:07.880 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 librados: starting msgr at 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 librados: starting objecter 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 Processor -- start 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 -- start start 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 librados: setting wanted keys 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 librados: calling monclient init 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 10 monclient: init 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 10 monclient(hunting): picked mon.noname-a con 0x7f87180a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 10 monclient(hunting): picked mon.noname-c con 0x7f871813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:07.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 10 monclient(hunting): picked mon.noname-b con 0x7f87181430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:07.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87180b7510 con 0x7f87180a98f0 2024-03-25T19:34:07.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87180b7c80 con 0x7f87181430c0 2024-03-25T19:34:07.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871812d970 con 0x7f871813f710 2024-03-25T19:34:07.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:07.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.877+0000 7f8720358640 10 monclient(hunting): authenticate will time out at 6264.222622s 2024-03-25T19:34:07.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.891+0000 7f871e8f5640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f87181430c0 0x7f871812d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:40952/0 (socket says 172.21.15.143:40952) 2024-03-25T19:34:07.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.891+0000 7f871e8f5640 1 -- 172.21.15.143:0/1066718569 learned_addr learned my addr 172.21.15.143:0/1066718569 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:07.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.894+0000 7f870effd640 10 client.?.objecter ms_handle_connect 0x7f87181430c0 2024-03-25T19:34:07.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.894+0000 7f870effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:07.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.896+0000 7f870effd640 10 client.?.objecter ms_handle_connect 0x7f871813f710 2024-03-25T19:34:07.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.896+0000 7f870effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:07.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.898+0000 7f870effd640 10 client.?.objecter ms_handle_connect 0x7f87180a98f0 2024-03-25T19:34:07.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.898+0000 7f870effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:07.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.908+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248954432 0 0) 0x7f87180b7c80 con 0x7f87181430c0 2024-03-25T19:34:07.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:07.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:07.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:07.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:34:07.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc004eb0 con 0x7f87181430c0 2024-03-25T19:34:07.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209088385 0 0) 0x7f871812d970 con 0x7f871813f710 2024-03-25T19:34:07.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:07.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:07.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:07.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:34:07.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.909+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87180b7c80 con 0x7f871813f710 2024-03-25T19:34:07.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.913+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962778249 0 0) 0x7f87180b7510 con 0x7f87180a98f0 2024-03-25T19:34:07.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.913+0000 7f870effd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:07.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.913+0000 7f870effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:07.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.913+0000 7f870effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:07.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.913+0000 7f870effd640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:34:07.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.913+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871812d970 con 0x7f87180a98f0 2024-03-25T19:34:07.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.923+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217649028 0 0) 0x7f87180b7c80 con 0x7f871813f710 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.923+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87180b7510 con 0x7f871813f710 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.923+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f87080014b0 con 0x7f871813f710 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.923+0000 7f870effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.923+0000 7f870effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.923+0000 7f870effd640 10 monclient(hunting): dump: 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:07.924 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:07.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.923+0000 7f870effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:07.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.934+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1455977204 0 0) 0x7f86fc004eb0 con 0x7f87181430c0 2024-03-25T19:34:07.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.934+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87180b7c80 con 0x7f87181430c0 2024-03-25T19:34:07.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.934+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8710001c30 con 0x7f87181430c0 2024-03-25T19:34:07.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.934+0000 7f870effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.934+0000 7f870effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.934+0000 7f870effd640 10 monclient(hunting): dump: 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:07.936 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:07.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2335143616 0 0) 0x7f87180b7510 con 0x7f871813f710 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 >> v1:172.21.15.203:6789/0 conn(0x7f87181430c0 legacy=0x7f871812d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 >> v1:172.21.15.143:6789/0 conn(0x7f87180a98f0 legacy=0x7f8718139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 1 monclient: found mon.c 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871813be70 con 0x7f871813f710 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 10 monclient: _finish_auth 0 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f870effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 5 monclient: authenticate success, global_id 4403 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 10 monclient: _renew_subs 2024-03-25T19:34:07.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:07.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871813b490 con 0x7f871813f710 2024-03-25T19:34:07.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:07.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 10 monclient: _renew_subs 2024-03-25T19:34:07.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:07.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f871813ba50 con 0x7f871813f710 2024-03-25T19:34:07.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.940+0000 7f8720358640 1 librados: init done 2024-03-25T19:34:07.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.942+0000 7f8720358640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:07.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.942+0000 7f8720358640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:07.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.942+0000 7f8720358640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:07.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:07.942+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871814f2d0 con 0x7f871813f710 2024-03-25T19:34:08.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.252+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8708001270 con 0x7f871813f710 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.252+0000 7f870effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.252+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8708004c30 con 0x7f871813f710 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.252+0000 7f870effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.252+0000 7f870effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.252+0000 7f870effd640 10 monclient: dump: 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:08.253 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:08.254 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:08.254 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:08.254 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:08.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.252+0000 7f871c8f1640 4 set_mon_vals no callback set 2024-03-25T19:34:08.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.253+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f870801ac80 con 0x7f871813f710 2024-03-25T19:34:08.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.253+0000 7f871e0f4640 10 monclient: get_auth_request con 0x7f86fc06e110 auth_method 0 2024-03-25T19:34:08.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.253+0000 7f870effd640 10 client.4403.objecter ms_handle_connect 0x7f86fc06e110 2024-03-25T19:34:08.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.258+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f8708054870 con 0x7f871813f710 2024-03-25T19:34:08.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.258+0000 7f870effd640 10 client.4403.objecter ms_dispatch 0x7f8718000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:08.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.258+0000 7f870effd640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:08.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.258+0000 7f870effd640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:08.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.258+0000 7f870effd640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-25T19:34:08.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.259+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87080501b0 con 0x7f871813f710 2024-03-25T19:34:08.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.259+0000 7f870effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:08.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.259+0000 7f870effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:08.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.517+0000 7f8720358640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:08.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.517+0000 7f8720358640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:08.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.517+0000 7f8720358640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:08.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.517+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f871813d640 con 0x7f871813f710 2024-03-25T19:34:08.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.543+0000 7f870effd640 1 -- 172.21.15.143:0/1066718569 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f87080522b0 con 0x7f871813f710 2024-03-25T19:34:08.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.543+0000 7f870effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:08.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.543+0000 7f870effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:08.544 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:34:08.544 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:08.546 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.545+0000 7f8720358640 10 librados: watch_flush enter 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.545+0000 7f8720358640 10 librados: watch_flush exit 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.545+0000 7f8720358640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.545+0000 7f8720358640 10 client.4403.objecter successfully canceled tick 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.545+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 >> v1:172.21.15.143:6800/1292282666 conn(0x7f86fc06e110 legacy=0x7f86fc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.545+0000 7f8720358640 10 monclient: shutdown 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.545+0000 7f8720358640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.545+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 >> v1:172.21.15.143:6790/0 conn(0x7f871813f710 legacy=0x7f871812cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.546+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 shutdown_connections 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.546+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 >> 172.21.15.143:0/1066718569 conn(0x7f87180b7ea0 msgr2=0x7f8718074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.546+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 shutdown_connections 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.546+0000 7f8720358640 1 -- 172.21.15.143:0/1066718569 wait complete. 2024-03-25T19:34:08.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:08.546+0000 7f8720358640 1 librados: shutdown 2024-03-25T19:34:08.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:13.475 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:04:40.039 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:13.481 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:04:40.046 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:13.525 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:04:40.090 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:13.539 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:40.075 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:13.548 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:04:40.113 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:13.549 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:40.084 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:13.552 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:04:40.117 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:13.568 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:04:40.104 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:13.583 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:04:40.148 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:15.365 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:15.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.479+0000 7f52f33ab640 10 monclient: get_monmap_and_config 2024-03-25T19:34:15.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.479+0000 7f52f33ab640 10 monclient: build_initial_monmap 2024-03-25T19:34:15.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.479+0000 7f52f33ab640 10 monclient: monmap: 2024-03-25T19:34:15.479 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:15.479 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:15.479 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:34:15.480391+0000 2024-03-25T19:34:15.479 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:34:15.480391+0000 2024-03-25T19:34:15.479 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:15.480 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:15.480 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:15.480 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:34:15.480 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:34:15.480 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.480+0000 7f52f33ab640 1 Processor -- start 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.480+0000 7f52f33ab640 1 -- start start 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.480+0000 7f52f33ab640 10 monclient: init 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 10 monclient(hunting): picked mon.noname-a con 0x7f52ec0a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 10 monclient(hunting): picked mon.noname-b con 0x7f52ec13f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 10 monclient(hunting): picked mon.noname-c con 0x7f52ec1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:15.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec142520 con 0x7f52ec0a98f0 2024-03-25T19:34:15.482 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0b28a0 con 0x7f52ec1430c0 2024-03-25T19:34:15.482 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0b2a30 con 0x7f52ec13f710 2024-03-25T19:34:15.482 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:15.482 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.481+0000 7f52f33ab640 10 monclient(hunting): authenticate will time out at 6271.825112s 2024-03-25T19:34:15.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.493+0000 7f52f0946640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f52ec13f710 0x7f52ec1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:50848/0 (socket says 172.21.15.143:50848) 2024-03-25T19:34:15.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.493+0000 7f52f0946640 1 -- 172.21.15.143:0/24531364 learned_addr learned my addr 172.21.15.143:0/24531364 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:15.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.499+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059591210 0 0) 0x7f52ec142520 con 0x7f52ec0a98f0 2024-03-25T19:34:15.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.499+0000 7f52e3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:15.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.499+0000 7f52e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:15.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.499+0000 7f52e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:15.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.499+0000 7f52e3fff640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:34:15.499 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.499+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d8003520 con 0x7f52ec0a98f0 2024-03-25T19:34:15.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.507+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373222714 0 0) 0x7f52ec0b28a0 con 0x7f52ec1430c0 2024-03-25T19:34:15.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.507+0000 7f52e3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:34:15.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.507+0000 7f52e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:15.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.507+0000 7f52e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:15.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.507+0000 7f52e3fff640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:34:15.507 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.507+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ec142520 con 0x7f52ec1430c0 2024-03-25T19:34:15.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1875462826 0 0) 0x7f52d8003520 con 0x7f52ec0a98f0 2024-03-25T19:34:15.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 >> v1:172.21.15.143:6790/0 conn(0x7f52ec13f710 legacy=0x7f52ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:15.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 >> v1:172.21.15.203:6789/0 conn(0x7f52ec1430c0 legacy=0x7f52ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:15.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:34:15.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 1 monclient: found mon.noname-a 2024-03-25T19:34:15.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0b44c0 con 0x7f52ec0a98f0 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 10 monclient: _finish_auth 0 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f52e8001690 con 0x7f52ec0a98f0 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52e3fff640 10 monclient: dump: 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:15.520 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:15.521 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:15.521 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:15.521 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:15.521 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52f33ab640 5 monclient: authenticate success, global_id 4413 2024-03-25T19:34:15.521 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.519+0000 7f52f33ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:15.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.595+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52e80017f0 con 0x7f52ec0a98f0 2024-03-25T19:34:15.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.595+0000 7f52e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:15.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.595+0000 7f52e3fff640 1 -- 172.21.15.143:0/24531364 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f52e8001c00 con 0x7f52ec0a98f0 2024-03-25T19:34:15.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.595+0000 7f52f33ab640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:15.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.595+0000 7f52f33ab640 4 set_mon_vals no callback set 2024-03-25T19:34:15.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.595+0000 7f52e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:15.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.595+0000 7f52e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.595+0000 7f52e3fff640 10 monclient: dump: 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:15.596 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.596+0000 7f52f33ab640 10 monclient: shutdown 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.596+0000 7f52f33ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.596+0000 7f52f33ab640 1 -- 172.21.15.143:0/24531364 >> v1:172.21.15.143:6789/0 conn(0x7f52ec0a98f0 legacy=0x7f52ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.596+0000 7f52f33ab640 1 -- 172.21.15.143:0/24531364 shutdown_connections 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.596+0000 7f52f33ab640 1 -- 172.21.15.143:0/24531364 >> 172.21.15.143:0/24531364 conn(0x7f52ec0b7ea0 msgr2=0x7f52ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.596+0000 7f52f33ab640 1 -- 172.21.15.143:0/24531364 shutdown_connections 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.596+0000 7f52f33ab640 1 -- 172.21.15.143:0/24531364 wait complete. 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 10 monclient: build_initial_monmap 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 10 monclient: monmap: 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:15.598 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 1 librados: starting msgr at 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 1 librados: starting objecter 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 1 Processor -- start 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 1 -- start start 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 1 librados: setting wanted keys 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 1 librados: calling monclient init 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.597+0000 7f52f33ab640 10 monclient: init 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:15.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:15.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 10 monclient(hunting): picked mon.noname-b con 0x7f52ec0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:15.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 10 monclient(hunting): picked mon.noname-c con 0x7f52ec13f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:15.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 10 monclient(hunting): picked mon.noname-a con 0x7f52ec1430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:15.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec132a30 con 0x7f52ec1430c0 2024-03-25T19:34:15.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec138c80 con 0x7f52ec0a98f0 2024-03-25T19:34:15.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec138f60 con 0x7f52ec13f710 2024-03-25T19:34:15.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:15.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.598+0000 7f52f33ab640 10 monclient(hunting): authenticate will time out at 6271.942124s 2024-03-25T19:34:15.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.616+0000 7f52f1948640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f52ec1430c0 0x7f52ec132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:41824/0 (socket says 172.21.15.143:41824) 2024-03-25T19:34:15.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.616+0000 7f52f1948640 1 -- 172.21.15.143:0/1477238971 learned_addr learned my addr 172.21.15.143:0/1477238971 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:15.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.617+0000 7f52e1ffb640 10 client.?.objecter ms_handle_connect 0x7f52ec1430c0 2024-03-25T19:34:15.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.617+0000 7f52e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:15.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.625+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211024633 0 0) 0x7f52ec132a30 con 0x7f52ec1430c0 2024-03-25T19:34:15.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.626+0000 7f52e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:15.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.626+0000 7f52e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:15.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.626+0000 7f52e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:15.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.626+0000 7f52e1ffb640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:34:15.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.626+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0004eb0 con 0x7f52ec1430c0 2024-03-25T19:34:15.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.629+0000 7f52e1ffb640 10 client.?.objecter ms_handle_connect 0x7f52ec13f710 2024-03-25T19:34:15.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.629+0000 7f52e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:15.636 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.637+0000 7f52e1ffb640 10 client.?.objecter ms_handle_connect 0x7f52ec0a98f0 2024-03-25T19:34:15.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.637+0000 7f52e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:15.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2299830348 0 0) 0x7f52d0004eb0 con 0x7f52ec1430c0 2024-03-25T19:34:15.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52ec132a30 con 0x7f52ec1430c0 2024-03-25T19:34:15.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f52e4001c90 con 0x7f52ec1430c0 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:15.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513189883 0 0) 0x7f52ec138f60 con 0x7f52ec13f710 2024-03-25T19:34:15.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:15.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:15.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:15.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:34:15.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.643+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0004eb0 con 0x7f52ec13f710 2024-03-25T19:34:15.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.647+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305890135 0 0) 0x7f52ec138c80 con 0x7f52ec0a98f0 2024-03-25T19:34:15.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.647+0000 7f52e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:15.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.647+0000 7f52e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:15.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.647+0000 7f52e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:15.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.647+0000 7f52e1ffb640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:34:15.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.647+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ec138f60 con 0x7f52ec0a98f0 2024-03-25T19:34:15.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.652+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1155533438 0 0) 0x7f52d0004eb0 con 0x7f52ec13f710 2024-03-25T19:34:15.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.652+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52ec138c80 con 0x7f52ec13f710 2024-03-25T19:34:15.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.652+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f52dc001860 con 0x7f52ec13f710 2024-03-25T19:34:15.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.652+0000 7f52e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:15.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.652+0000 7f52e1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:15.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.652+0000 7f52e1ffb640 10 monclient(hunting): dump: 2024-03-25T19:34:15.652 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:15.652 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:15.653 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.653 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.653 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:15.653 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:15.653 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:15.653 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:15.653 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:15.653 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:15.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 828370654 0 0) 0x7f52ec132a30 con 0x7f52ec1430c0 2024-03-25T19:34:15.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 >> v1:172.21.15.143:6790/0 conn(0x7f52ec13f710 legacy=0x7f52ec131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:15.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 >> v1:172.21.15.203:6789/0 conn(0x7f52ec0a98f0 legacy=0x7f52ec137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:15.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 1 monclient: found mon.a 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec139240 con 0x7f52ec1430c0 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 5 monclient: authenticate success, global_id 4416 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 10 monclient: _renew_subs 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec136e50 con 0x7f52ec1430c0 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 10 monclient: _renew_subs 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec1389f0 con 0x7f52ec1430c0 2024-03-25T19:34:15.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.653+0000 7f52f33ab640 1 librados: init done 2024-03-25T19:34:15.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.654+0000 7f52f33ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:15.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.654+0000 7f52f33ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:15.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.654+0000 7f52f33ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:15.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.654+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec0627b0 con 0x7f52ec1430c0 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.940+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52e4003970 con 0x7f52ec1430c0 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.940+0000 7f52e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.940+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f52e4003d00 con 0x7f52ec1430c0 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.940+0000 7f52e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.940+0000 7f52e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.940+0000 7f52e1ffb640 10 monclient: dump: 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:15.940 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:15.941 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:15.941 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:15.941 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:15.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.940+0000 7f52e37fe640 4 set_mon_vals no callback set 2024-03-25T19:34:15.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.941+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f52e4003f20 con 0x7f52ec1430c0 2024-03-25T19:34:15.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.942+0000 7f52f1147640 10 monclient: get_auth_request con 0x7f52d006dfd0 auth_method 0 2024-03-25T19:34:15.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.942+0000 7f52e1ffb640 10 client.4416.objecter ms_handle_connect 0x7f52d006dfd0 2024-03-25T19:34:15.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.949+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f52e4054ae0 con 0x7f52ec1430c0 2024-03-25T19:34:15.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.949+0000 7f52e1ffb640 10 client.4416.objecter ms_dispatch 0x7f52ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:15.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.949+0000 7f52e1ffb640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:15.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.949+0000 7f52e1ffb640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:15.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.949+0000 7f52e1ffb640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-25T19:34:15.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.949+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52e40505f0 con 0x7f52ec1430c0 2024-03-25T19:34:15.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.949+0000 7f52e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:15.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:15.949+0000 7f52e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:16.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.198+0000 7f52f33ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:16.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.198+0000 7f52f33ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:16.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.198+0000 7f52f33ab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:16.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.198+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52ec13dd60 con 0x7f52ec1430c0 2024-03-25T19:34:16.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.228+0000 7f52e1ffb640 1 -- 172.21.15.143:0/1477238971 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f52e4052520 con 0x7f52ec1430c0 2024-03-25T19:34:16.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.228+0000 7f52e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:16.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.228+0000 7f52e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:16.228 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:34:16.228 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 10 librados: watch_flush enter 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 10 librados: watch_flush exit 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 10 client.4416.objecter successfully canceled tick 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 >> v1:172.21.15.143:6800/1292282666 conn(0x7f52d006dfd0 legacy=0x7f52d0070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 10 monclient: shutdown 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 >> v1:172.21.15.143:6789/0 conn(0x7f52ec1430c0 legacy=0x7f52ec132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:16.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 shutdown_connections 2024-03-25T19:34:16.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 >> 172.21.15.143:0/1477238971 conn(0x7f52ec0b7ea0 msgr2=0x7f52ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:16.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 shutdown_connections 2024-03-25T19:34:16.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 1 -- 172.21.15.143:0/1477238971 wait complete. 2024-03-25T19:34:16.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:16.231+0000 7f52f33ab640 1 librados: shutdown 2024-03-25T19:34:16.243 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:23.050 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.165+0000 7fc07ebc1640 10 monclient: get_monmap_and_config 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.165+0000 7fc07ebc1640 10 monclient: build_initial_monmap 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.165+0000 7fc07ebc1640 10 monclient: monmap: 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:34:23.166452+0000 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:34:23.166452+0000 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:23.165 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:23.166 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:23.166 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:34:23.166 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:34:23.166 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:23.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.166+0000 7fc07ebc1640 1 Processor -- start 2024-03-25T19:34:23.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.166+0000 7fc07ebc1640 1 -- start start 2024-03-25T19:34:23.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.166+0000 7fc07ebc1640 10 monclient: init 2024-03-25T19:34:23.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:23.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:23.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 10 monclient(hunting): picked mon.noname-a con 0x7fc0780a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:23.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 10 monclient(hunting): picked mon.noname-c con 0x7fc07813f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:23.167 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 10 monclient(hunting): picked mon.noname-b con 0x7fc0781430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:23.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc078142520 con 0x7fc0780a98f0 2024-03-25T19:34:23.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0780b28a0 con 0x7fc07813f710 2024-03-25T19:34:23.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0780b2a30 con 0x7fc0781430c0 2024-03-25T19:34:23.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:23.168 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.167+0000 7fc07ebc1640 10 monclient(hunting): authenticate will time out at 6279.511208s 2024-03-25T19:34:23.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.180+0000 7fc077fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc07813f710 0x7fc0781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:35770/0 (socket says 172.21.15.143:35770) 2024-03-25T19:34:23.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.180+0000 7fc077fff640 1 -- 172.21.15.143:0/3836749431 learned_addr learned my addr 172.21.15.143:0/3836749431 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:23.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.204+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443215802 0 0) 0x7fc0780b28a0 con 0x7fc07813f710 2024-03-25T19:34:23.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.204+0000 7fc0777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:23.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.204+0000 7fc0777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:23.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.204+0000 7fc0777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:23.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.204+0000 7fc0777fe640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:34:23.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.204+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc068003520 con 0x7fc07813f710 2024-03-25T19:34:23.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.208+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900228772 0 0) 0x7fc078142520 con 0x7fc0780a98f0 2024-03-25T19:34:23.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.208+0000 7fc0777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:23.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.208+0000 7fc0777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:23.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.208+0000 7fc0777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:23.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.208+0000 7fc0777fe640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:34:23.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.208+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0780b28a0 con 0x7fc0780a98f0 2024-03-25T19:34:23.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.214+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426345545 0 0) 0x7fc0780b2a30 con 0x7fc0781430c0 2024-03-25T19:34:23.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.214+0000 7fc0777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:23.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.214+0000 7fc0777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:23.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.214+0000 7fc0777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:23.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.214+0000 7fc0777fe640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:34:23.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.214+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc078142520 con 0x7fc0781430c0 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 741068229 0 0) 0x7fc078142520 con 0x7fc0781430c0 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 >> v1:172.21.15.203:6789/0 conn(0x7fc07813f710 legacy=0x7fc0781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 >> v1:172.21.15.143:6789/0 conn(0x7fc0780a98f0 legacy=0x7fc0780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 1 monclient: found mon.noname-b 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0780b44c0 con 0x7fc0781430c0 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc070001930 con 0x7fc0781430c0 2024-03-25T19:34:23.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc07ebc1640 5 monclient: authenticate success, global_id 4409 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc07ebc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc0777fe640 10 monclient: dump: 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:23.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.230+0000 7fc07ebc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.247+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc070001a90 con 0x7fc0781430c0 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.247+0000 7fc0777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.247+0000 7fc0777fe640 1 -- 172.21.15.143:0/3836749431 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc070001ee0 con 0x7fc0781430c0 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.247+0000 7fc07ebc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.247+0000 7fc07ebc1640 4 set_mon_vals no callback set 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.247+0000 7fc0777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.247+0000 7fc0777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.247+0000 7fc0777fe640 10 monclient: dump: 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:23.247 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:23.248 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:23.248 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:23.248 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:23.248 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:23.248 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:23.248 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:23.248 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:23.248 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:23.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.248+0000 7fc07ebc1640 10 monclient: shutdown 2024-03-25T19:34:23.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.248+0000 7fc07ebc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:23.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.248+0000 7fc07ebc1640 1 -- 172.21.15.143:0/3836749431 >> v1:172.21.15.143:6790/0 conn(0x7fc0781430c0 legacy=0x7fc07814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:23.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.248+0000 7fc07ebc1640 1 -- 172.21.15.143:0/3836749431 shutdown_connections 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.248+0000 7fc07ebc1640 1 -- 172.21.15.143:0/3836749431 >> 172.21.15.143:0/3836749431 conn(0x7fc0780b7ea0 msgr2=0x7fc0780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.248+0000 7fc07ebc1640 1 -- 172.21.15.143:0/3836749431 shutdown_connections 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.248+0000 7fc07ebc1640 1 -- 172.21.15.143:0/3836749431 wait complete. 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 10 monclient: build_initial_monmap 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 10 monclient: monmap: 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:34:23.250 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:23.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 1 librados: starting msgr at 2024-03-25T19:34:23.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 1 librados: starting objecter 2024-03-25T19:34:23.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 1 Processor -- start 2024-03-25T19:34:23.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 1 -- start start 2024-03-25T19:34:23.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 1 librados: setting wanted keys 2024-03-25T19:34:23.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 1 librados: calling monclient init 2024-03-25T19:34:23.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.249+0000 7fc07ebc1640 10 monclient: init 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 10 monclient(hunting): picked mon.noname-a con 0x7fc0780a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 10 monclient(hunting): picked mon.noname-c con 0x7fc07813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 10 monclient(hunting): picked mon.noname-b con 0x7fc0781430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc078139d20 con 0x7fc0780a98f0 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07813ad50 con 0x7fc0781430c0 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07813b030 con 0x7fc07813f710 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:23.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.250+0000 7fc07ebc1640 10 monclient(hunting): authenticate will time out at 6279.594418s 2024-03-25T19:34:23.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.270+0000 7fc077fff640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fc07813f710 0x7fc07813ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:37762/0 (socket says 172.21.15.143:37762) 2024-03-25T19:34:23.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.270+0000 7fc077fff640 1 -- 172.21.15.143:0/1509956218 learned_addr learned my addr 172.21.15.143:0/1509956218 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:23.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.273+0000 7fc0757fa640 10 client.?.objecter ms_handle_connect 0x7fc07813f710 2024-03-25T19:34:23.272 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.273+0000 7fc0757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:23.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.274+0000 7fc0757fa640 10 client.?.objecter ms_handle_connect 0x7fc0781430c0 2024-03-25T19:34:23.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.274+0000 7fc0757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:23.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.277+0000 7fc0757fa640 10 client.?.objecter ms_handle_connect 0x7fc0780a98f0 2024-03-25T19:34:23.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.277+0000 7fc0757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:23.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.278+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107159327 0 0) 0x7fc07813ad50 con 0x7fc0781430c0 2024-03-25T19:34:23.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.278+0000 7fc0757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:23.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.278+0000 7fc0757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:23.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.278+0000 7fc0757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:23.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.278+0000 7fc0757fa640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:34:23.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.278+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c004eb0 con 0x7fc0781430c0 2024-03-25T19:34:23.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.286+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520474996 0 0) 0x7fc07813b030 con 0x7fc07813f710 2024-03-25T19:34:23.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.286+0000 7fc0757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:23.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.286+0000 7fc0757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:23.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.286+0000 7fc0757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:23.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.286+0000 7fc0757fa640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:34:23.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.286+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc07813ad50 con 0x7fc07813f710 2024-03-25T19:34:23.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.291+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125259285 0 0) 0x7fc078139d20 con 0x7fc0780a98f0 2024-03-25T19:34:23.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.291+0000 7fc0757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:23.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.291+0000 7fc0757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:23.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.291+0000 7fc0757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:23.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.291+0000 7fc0757fa640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:34:23.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.291+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc07813b030 con 0x7fc0780a98f0 2024-03-25T19:34:23.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.295+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3536581239 0 0) 0x7fc05c004eb0 con 0x7fc0781430c0 2024-03-25T19:34:23.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.295+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc078139d20 con 0x7fc0781430c0 2024-03-25T19:34:23.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.295+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc070001de0 con 0x7fc0781430c0 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.295+0000 7fc0757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.295+0000 7fc0757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.295+0000 7fc0757fa640 10 monclient(hunting): dump: 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:23.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.295+0000 7fc0757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:23.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2141475089 0 0) 0x7fc078139d20 con 0x7fc0781430c0 2024-03-25T19:34:23.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 >> v1:172.21.15.143:6790/0 conn(0x7fc07813f710 legacy=0x7fc07813ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:23.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 >> v1:172.21.15.143:6789/0 conn(0x7fc0780a98f0 legacy=0x7fc07812f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:23.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:23.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 1 monclient: found mon.b 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07813b310 con 0x7fc0781430c0 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc0757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 5 monclient: authenticate success, global_id 4423 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 10 monclient: _renew_subs 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07813b5f0 con 0x7fc0781430c0 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 10 monclient: _renew_subs 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07813c300 con 0x7fc0781430c0 2024-03-25T19:34:23.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.308+0000 7fc07ebc1640 1 librados: init done 2024-03-25T19:34:23.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.309+0000 7fc07ebc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:23.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.309+0000 7fc07ebc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:23.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.309+0000 7fc07ebc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:23.310 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.309+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0780627b0 con 0x7fc0781430c0 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.605+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc070002020 con 0x7fc0781430c0 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.605+0000 7fc0757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.605+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc070003dc0 con 0x7fc0781430c0 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.605+0000 7fc0757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.605+0000 7fc0757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.605+0000 7fc0757fa640 10 monclient: dump: 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:23.605 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.605+0000 7fc076ffd640 4 set_mon_vals no callback set 2024-03-25T19:34:23.606 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.606+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fc070019d90 con 0x7fc0781430c0 2024-03-25T19:34:23.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.607+0000 7fc07c95d640 10 monclient: get_auth_request con 0x7fc05c06ddf0 auth_method 0 2024-03-25T19:34:23.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.607+0000 7fc0757fa640 10 client.4423.objecter ms_handle_connect 0x7fc05c06ddf0 2024-03-25T19:34:23.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.611+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fc070053900 con 0x7fc0781430c0 2024-03-25T19:34:23.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.611+0000 7fc0757fa640 10 client.4423.objecter ms_dispatch 0x7fc078000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:23.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.611+0000 7fc0757fa640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:23.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.611+0000 7fc0757fa640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:23.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.611+0000 7fc0757fa640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-25T19:34:23.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.611+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc07004f410 con 0x7fc0781430c0 2024-03-25T19:34:23.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.611+0000 7fc0757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:23.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.611+0000 7fc0757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:23.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.833+0000 7fc07ebc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:23.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.833+0000 7fc07ebc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:23.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.833+0000 7fc07ebc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:23.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.833+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc07813de20 con 0x7fc0781430c0 2024-03-25T19:34:23.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.865+0000 7fc0757fa640 1 -- 172.21.15.143:0/1509956218 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fc070051340 con 0x7fc0781430c0 2024-03-25T19:34:23.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.865+0000 7fc0757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:23.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.865+0000 7fc0757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:23.865 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:34:23.865 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 10 librados: watch_flush enter 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 10 librados: watch_flush exit 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 10 client.4423.objecter successfully canceled tick 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 >> v1:172.21.15.143:6800/1292282666 conn(0x7fc05c06ddf0 legacy=0x7fc05c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 10 monclient: shutdown 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 >> v1:172.21.15.203:6789/0 conn(0x7fc0781430c0 legacy=0x7fc0781e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 shutdown_connections 2024-03-25T19:34:23.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 >> 172.21.15.143:0/1509956218 conn(0x7fc0780b7ea0 msgr2=0x7fc078074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:23.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 shutdown_connections 2024-03-25T19:34:23.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 1 -- 172.21.15.143:0/1509956218 wait complete. 2024-03-25T19:34:23.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:23.868+0000 7fc07ebc1640 1 librados: shutdown 2024-03-25T19:34:23.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:30.686 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.796+0000 7fcbbd4fb640 10 monclient: get_monmap_and_config 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.796+0000 7fcbbd4fb640 10 monclient: build_initial_monmap 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.796+0000 7fcbbd4fb640 10 monclient: monmap: 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:34:30.797684+0000 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:34:30.797684+0000 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:30.796 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:30.797 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:34:30.797 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:34:30.797 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:30.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.797+0000 7fcbbd4fb640 1 Processor -- start 2024-03-25T19:34:30.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.797+0000 7fcbbd4fb640 1 -- start start 2024-03-25T19:34:30.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.797+0000 7fcbbd4fb640 10 monclient: init 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 10 monclient(hunting): picked mon.noname-b con 0x7fcbb80a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 10 monclient(hunting): picked mon.noname-a con 0x7fcbb813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 10 monclient(hunting): picked mon.noname-c con 0x7fcbb81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb8142520 con 0x7fcbb813f710 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb80b28a0 con 0x7fcbb81430c0 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb80b2a30 con 0x7fcbb80a98f0 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:30.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.798+0000 7fcbbd4fb640 10 monclient(hunting): authenticate will time out at 6287.142455s 2024-03-25T19:34:30.809 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.810+0000 7fcbb77fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fcbb81430c0 0x7fcbb814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:36272/0 (socket says 172.21.15.143:36272) 2024-03-25T19:34:30.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.810+0000 7fcbb77fe640 1 -- 172.21.15.143:0/2198483553 learned_addr learned my addr 172.21.15.143:0/2198483553 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:30.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.829+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277659384 0 0) 0x7fcbb80b28a0 con 0x7fcbb81430c0 2024-03-25T19:34:30.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.829+0000 7fcbb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:30.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.829+0000 7fcbb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:30.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.829+0000 7fcbb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:30.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.829+0000 7fcbb5ffb640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:34:30.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.829+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba0003520 con 0x7fcbb81430c0 2024-03-25T19:34:30.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.832+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035893468 0 0) 0x7fcbb8142520 con 0x7fcbb813f710 2024-03-25T19:34:30.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.832+0000 7fcbb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:30.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.832+0000 7fcbb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:30.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.832+0000 7fcbb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:30.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.832+0000 7fcbb5ffb640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:34:30.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.832+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb80b28a0 con 0x7fcbb813f710 2024-03-25T19:34:30.842 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.843+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713495833 0 0) 0x7fcbb80b2a30 con 0x7fcbb80a98f0 2024-03-25T19:34:30.842 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.843+0000 7fcbb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:34:30.842 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.843+0000 7fcbb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:30.842 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.843+0000 7fcbb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:30.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.843+0000 7fcbb5ffb640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:34:30.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.843+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb8142520 con 0x7fcbb80a98f0 2024-03-25T19:34:30.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2582588539 0 0) 0x7fcba0003520 con 0x7fcbb81430c0 2024-03-25T19:34:30.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 >> v1:172.21.15.143:6790/0 conn(0x7fcbb80a98f0 legacy=0x7fcbb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:30.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 >> v1:172.21.15.143:6789/0 conn(0x7fcbb813f710 legacy=0x7fcbb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:30.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:34:30.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:34:30.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:30.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb80b44c0 con 0x7fcbb81430c0 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fcbb0001a10 con 0x7fcbb81430c0 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbbd4fb640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbbd4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbb5ffb640 10 monclient: dump: 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:30.862 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:30.863 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:30.863 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:30.863 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:30.863 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:30.863 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:30.863 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:30.863 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:30.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.861+0000 7fcbbd4fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.878+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbb0001b70 con 0x7fcbb81430c0 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.878+0000 7fcbb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.878+0000 7fcbb5ffb640 1 -- 172.21.15.143:0/2198483553 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fcbb0001f80 con 0x7fcbb81430c0 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.878+0000 7fcbb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.878+0000 7fcbb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.878+0000 7fcbb5ffb640 10 monclient: dump: 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:30.878 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:30.879 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:30.879 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:30.879 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:30.879 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:30.879 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:30.879 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:30.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.878+0000 7fcbbd4fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:30.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.878+0000 7fcbbd4fb640 4 set_mon_vals no callback set 2024-03-25T19:34:30.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.879+0000 7fcbbd4fb640 10 monclient: shutdown 2024-03-25T19:34:30.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.879+0000 7fcbbd4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:30.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.879+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/2198483553 >> v1:172.21.15.203:6789/0 conn(0x7fcbb81430c0 legacy=0x7fcbb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:30.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.879+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/2198483553 shutdown_connections 2024-03-25T19:34:30.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.879+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/2198483553 >> 172.21.15.143:0/2198483553 conn(0x7fcbb80b7ea0 msgr2=0x7fcbb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.879+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/2198483553 shutdown_connections 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.879+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/2198483553 wait complete. 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 10 monclient: build_initial_monmap 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 10 monclient: monmap: 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 1 librados: starting msgr at 2024-03-25T19:34:30.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 1 librados: starting objecter 2024-03-25T19:34:30.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 1 Processor -- start 2024-03-25T19:34:30.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 1 -- start start 2024-03-25T19:34:30.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 1 librados: setting wanted keys 2024-03-25T19:34:30.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 1 librados: calling monclient init 2024-03-25T19:34:30.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.880+0000 7fcbbd4fb640 10 monclient: init 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 10 monclient(hunting): picked mon.noname-a con 0x7fcbb80a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 10 monclient(hunting): picked mon.noname-c con 0x7fcbb813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 10 monclient(hunting): picked mon.noname-b con 0x7fcbb81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb8136ec0 con 0x7fcbb80a98f0 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb8136c20 con 0x7fcbb81430c0 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb8137db0 con 0x7fcbb813f710 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:30.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.881+0000 7fcbbd4fb640 10 monclient(hunting): authenticate will time out at 6287.225238s 2024-03-25T19:34:30.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.897+0000 7fcbb67fc640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fcbb813f710 0x7fcbb80b0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:47836/0 (socket says 172.21.15.143:47836) 2024-03-25T19:34:30.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.897+0000 7fcbb67fc640 1 -- 172.21.15.143:0/1919497697 learned_addr learned my addr 172.21.15.143:0/1919497697 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:30.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.898+0000 7fcb9f7fe640 10 client.?.objecter ms_handle_connect 0x7fcbb813f710 2024-03-25T19:34:30.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.898+0000 7fcb9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:30.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.899+0000 7fcb9f7fe640 10 client.?.objecter ms_handle_connect 0x7fcbb80a98f0 2024-03-25T19:34:30.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.899+0000 7fcb9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:30.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.906+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929491160 0 0) 0x7fcbb8137db0 con 0x7fcbb813f710 2024-03-25T19:34:30.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.906+0000 7fcb9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:30.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.906+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:30.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.906+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:30.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.906+0000 7fcb9f7fe640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:34:30.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.906+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcba4004eb0 con 0x7fcbb813f710 2024-03-25T19:34:30.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.907+0000 7fcb9f7fe640 10 client.?.objecter ms_handle_connect 0x7fcbb81430c0 2024-03-25T19:34:30.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.907+0000 7fcb9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:30.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.919+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794791685 0 0) 0x7fcbb8136ec0 con 0x7fcbb80a98f0 2024-03-25T19:34:30.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.919+0000 7fcb9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:30.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.919+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:30.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.919+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:30.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.919+0000 7fcb9f7fe640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:34:30.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.919+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb8137db0 con 0x7fcbb80a98f0 2024-03-25T19:34:30.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.921+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988000745 0 0) 0x7fcbb8136c20 con 0x7fcbb81430c0 2024-03-25T19:34:30.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.921+0000 7fcb9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:30.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.921+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:30.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.921+0000 7fcb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:30.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.921+0000 7fcb9f7fe640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:34:30.922 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.921+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb8136ec0 con 0x7fcbb81430c0 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.924+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1859951741 0 0) 0x7fcba4004eb0 con 0x7fcbb813f710 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.924+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbb8136c20 con 0x7fcbb813f710 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.924+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fcbac001a30 con 0x7fcbb813f710 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.924+0000 7fcb9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.924+0000 7fcb9f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.924+0000 7fcb9f7fe640 10 monclient(hunting): dump: 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:30.924 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:30.925 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:30.925 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:30.925 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:30.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.924+0000 7fcb9f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:30.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3698490928 0 0) 0x7fcbb8136c20 con 0x7fcbb813f710 2024-03-25T19:34:30.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 >> v1:172.21.15.203:6789/0 conn(0x7fcbb81430c0 legacy=0x7fcbb81e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:30.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 >> v1:172.21.15.143:6789/0 conn(0x7fcbb80a98f0 legacy=0x7fcbb81376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:30.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:30.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 1 monclient: found mon.c 2024-03-25T19:34:30.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:30.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:30.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb8138b50 con 0x7fcbb813f710 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcb9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 5 monclient: authenticate success, global_id 4418 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 10 monclient: _renew_subs 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb8139830 con 0x7fcbb813f710 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 10 monclient: _renew_subs 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb81369b0 con 0x7fcbb813f710 2024-03-25T19:34:30.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.943+0000 7fcbbd4fb640 1 librados: init done 2024-03-25T19:34:30.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.944+0000 7fcbbd4fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:30.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.944+0000 7fcbbd4fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:30.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.944+0000 7fcbbd4fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:30.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:30.944+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb80627b0 con 0x7fcbb813f710 2024-03-25T19:34:31.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.239+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcbac001c70 con 0x7fcbb813f710 2024-03-25T19:34:31.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.239+0000 7fcb9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:31.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.239+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fcbac003a60 con 0x7fcbb813f710 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.239+0000 7fcb9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.239+0000 7fcb9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.239+0000 7fcb9f7fe640 10 monclient: dump: 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:31.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.239+0000 7fcbb57fa640 4 set_mon_vals no callback set 2024-03-25T19:34:31.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.241+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fcbac003c40 con 0x7fcbb813f710 2024-03-25T19:34:31.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.241+0000 7fcbb6ffd640 10 monclient: get_auth_request con 0x7fcba406ddf0 auth_method 0 2024-03-25T19:34:31.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.241+0000 7fcb9f7fe640 10 client.4418.objecter ms_handle_connect 0x7fcba406ddf0 2024-03-25T19:34:31.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.245+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fcbac0548d0 con 0x7fcbb813f710 2024-03-25T19:34:31.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.245+0000 7fcb9f7fe640 10 client.4418.objecter ms_dispatch 0x7fcbb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:31.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.245+0000 7fcb9f7fe640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:31.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.245+0000 7fcb9f7fe640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:31.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.245+0000 7fcb9f7fe640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-25T19:34:31.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.246+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcbac0503e0 con 0x7fcbb813f710 2024-03-25T19:34:31.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.246+0000 7fcb9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:31.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.246+0000 7fcb9f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:31.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.511+0000 7fcbbd4fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:31.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.511+0000 7fcbbd4fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:31.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.511+0000 7fcbbd4fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:31.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.511+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbb8069f40 con 0x7fcbb813f710 2024-03-25T19:34:31.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.683+0000 7fcb9f7fe640 1 -- 172.21.15.143:0/1919497697 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fcbac052310 con 0x7fcbb813f710 2024-03-25T19:34:31.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.683+0000 7fcb9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:31.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.683+0000 7fcb9f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:31.683 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:34:31.683 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.685+0000 7fcbbd4fb640 10 librados: watch_flush enter 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.685+0000 7fcbbd4fb640 10 librados: watch_flush exit 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.685+0000 7fcbbd4fb640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.685+0000 7fcbbd4fb640 10 client.4418.objecter successfully canceled tick 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 >> v1:172.21.15.143:6800/1292282666 conn(0x7fcba406ddf0 legacy=0x7fcba40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 10 monclient: shutdown 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 >> v1:172.21.15.143:6790/0 conn(0x7fcbb813f710 legacy=0x7fcbb80b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:31.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 shutdown_connections 2024-03-25T19:34:31.687 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 >> 172.21.15.143:0/1919497697 conn(0x7fcbb80b7ea0 msgr2=0x7fcbb813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:31.687 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 shutdown_connections 2024-03-25T19:34:31.687 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 1 -- 172.21.15.143:0/1919497697 wait complete. 2024-03-25T19:34:31.687 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:31.686+0000 7fcbbd4fb640 1 librados: shutdown 2024-03-25T19:34:31.699 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:33.498 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:00.063 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:33.550 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:00.115 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:33.593 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:00.158 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:33.618 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:00.154 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:33.621 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:00.157 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:33.626 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:00.191 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:33.632 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:00.197 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:33.653 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:00.189 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:33.662 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:00.227 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:38.504 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.624+0000 7f6d7a75d640 10 monclient: get_monmap_and_config 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.624+0000 7f6d7a75d640 10 monclient: build_initial_monmap 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.624+0000 7f6d7a75d640 10 monclient: monmap: 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:34:38.625520+0000 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:34:38.625520+0000 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:38.624 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:34:38.625 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:34:38.625 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.625+0000 7f6d7a75d640 1 Processor -- start 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.625+0000 7f6d7a75d640 1 -- start start 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.625+0000 7f6d7a75d640 10 monclient: init 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d740b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d740b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d7414be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740af940 con 0x7f6d7414be90 2024-03-25T19:34:38.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740b0a70 con 0x7f6d740b4d60 2024-03-25T19:34:38.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740b0d50 con 0x7f6d740b1730 2024-03-25T19:34:38.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:38.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.626+0000 7f6d7a75d640 10 monclient(hunting): authenticate will time out at 6294.970047s 2024-03-25T19:34:38.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.636+0000 7f6d73fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6d740b4d60 0x7f6d7413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:55318/0 (socket says 172.21.15.143:55318) 2024-03-25T19:34:38.638 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.636+0000 7f6d73fff640 1 -- 172.21.15.143:0/3977498306 learned_addr learned my addr 172.21.15.143:0/3977498306 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:38.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.654+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798757460 0 0) 0x7f6d740b0a70 con 0x7f6d740b4d60 2024-03-25T19:34:38.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.654+0000 7f6d72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:38.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.654+0000 7f6d72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:38.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.654+0000 7f6d72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:38.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.654+0000 7f6d72ffd640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:34:38.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.654+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d5c003520 con 0x7f6d740b4d60 2024-03-25T19:34:38.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.659+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668077606 0 0) 0x7f6d740b0d50 con 0x7f6d740b1730 2024-03-25T19:34:38.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.659+0000 7f6d72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:38.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.659+0000 7f6d72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:38.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.659+0000 7f6d72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:38.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.659+0000 7f6d72ffd640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:34:38.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.659+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d740b0a70 con 0x7f6d740b1730 2024-03-25T19:34:38.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.666+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574649267 0 0) 0x7f6d740af940 con 0x7f6d7414be90 2024-03-25T19:34:38.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.666+0000 7f6d72ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:34:38.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.666+0000 7f6d72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:38.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.666+0000 7f6d72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:38.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.666+0000 7f6d72ffd640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:34:38.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.666+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d740b0d50 con 0x7f6d7414be90 2024-03-25T19:34:38.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2405276490 0 0) 0x7f6d5c003520 con 0x7f6d740b4d60 2024-03-25T19:34:38.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 >> v1:172.21.15.143:6790/0 conn(0x7f6d740b1730 legacy=0x7f6d740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:38.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 >> v1:172.21.15.143:6789/0 conn(0x7f6d7414be90 legacy=0x7f6d7414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:38.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:34:38.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 1 monclient: found mon.noname-c 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d740b1030 con 0x7f6d740b4d60 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 10 monclient: _finish_auth 0 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d64001690 con 0x7f6d740b4d60 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d72ffd640 10 monclient: dump: 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:38.678 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:38.679 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:38.679 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:38.679 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:38.679 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:38.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d7a75d640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:34:38.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.677+0000 7f6d7a75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:38.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d640017f0 con 0x7f6d740b4d60 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d72ffd640 1 -- 172.21.15.143:0/3977498306 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d64001c00 con 0x7f6d740b4d60 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d7a75d640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d7a75d640 4 set_mon_vals no callback set 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d72ffd640 10 monclient: dump: 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:38.714 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:38.715 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d7a75d640 10 monclient: shutdown 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d7a75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.714+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3977498306 >> v1:172.21.15.203:6789/0 conn(0x7f6d740b4d60 legacy=0x7f6d7413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.715+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3977498306 shutdown_connections 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.715+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3977498306 >> 172.21.15.143:0/3977498306 conn(0x7f6d740617a0 msgr2=0x7f6d74061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.715+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3977498306 shutdown_connections 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.715+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3977498306 wait complete. 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 10 monclient: build_initial_monmap 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 10 monclient: monmap: 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:34:38.716 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:38.717 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:34:38.717 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:38.717 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:34:38.717 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:34:38.717 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:38.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 1 librados: starting msgr at 2024-03-25T19:34:38.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 1 librados: starting objecter 2024-03-25T19:34:38.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 1 Processor -- start 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 1 -- start start 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 1 librados: setting wanted keys 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 1 librados: calling monclient init 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.716+0000 7f6d7a75d640 10 monclient: init 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d740b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d740b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d7414be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d74140ce0 con 0x7f6d7414be90 2024-03-25T19:34:38.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d741d7a00 con 0x7f6d740b1730 2024-03-25T19:34:38.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d741d85a0 con 0x7f6d740b4d60 2024-03-25T19:34:38.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:38.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.717+0000 7f6d7a75d640 10 monclient(hunting): authenticate will time out at 6295.061431s 2024-03-25T19:34:38.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.725+0000 7f6d737fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f6d740b4d60 0x7f6d7413b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:58904/0 (socket says 172.21.15.143:58904) 2024-03-25T19:34:38.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.725+0000 7f6d737fe640 1 -- 172.21.15.143:0/3797852532 learned_addr learned my addr 172.21.15.143:0/3797852532 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:38.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.727+0000 7f6d70ff9640 10 client.?.objecter ms_handle_connect 0x7f6d740b4d60 2024-03-25T19:34:38.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.727+0000 7f6d70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:38.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.732+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154930294 0 0) 0x7f6d741d85a0 con 0x7f6d740b4d60 2024-03-25T19:34:38.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.732+0000 7f6d70ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:38.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.732+0000 7f6d70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:38.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.732+0000 7f6d70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:38.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.732+0000 7f6d70ff9640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:34:38.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.732+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d60004eb0 con 0x7f6d740b4d60 2024-03-25T19:34:38.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.734+0000 7f6d70ff9640 10 client.?.objecter ms_handle_connect 0x7f6d740b1730 2024-03-25T19:34:38.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.734+0000 7f6d70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:38.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.737+0000 7f6d70ff9640 10 client.?.objecter ms_handle_connect 0x7f6d7414be90 2024-03-25T19:34:38.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.737+0000 7f6d70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:38.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.741+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3899809446 0 0) 0x7f6d60004eb0 con 0x7f6d740b4d60 2024-03-25T19:34:38.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.741+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d741d85a0 con 0x7f6d740b4d60 2024-03-25T19:34:38.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.741+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d68001a30 con 0x7f6d740b4d60 2024-03-25T19:34:38.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.741+0000 7f6d70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:38.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.741+0000 7f6d70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:38.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.741+0000 7f6d70ff9640 10 monclient(hunting): dump: 2024-03-25T19:34:38.741 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:38.742 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.741+0000 7f6d70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:38.749 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.749+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2003953533 0 0) 0x7f6d741d85a0 con 0x7f6d740b4d60 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 >> v1:172.21.15.203:6789/0 conn(0x7f6d740b1730 legacy=0x7f6d741d7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 >> v1:172.21.15.143:6789/0 conn(0x7f6d7414be90 legacy=0x7f6d74142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 1 monclient: found mon.c 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d741d7690 con 0x7f6d740b4d60 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 10 monclient: _finish_auth 0 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 5 monclient: authenticate success, global_id 4424 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 10 monclient: _renew_subs 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:38.750 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7413bcc0 con 0x7f6d740b4d60 2024-03-25T19:34:38.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:38.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 10 monclient: _renew_subs 2024-03-25T19:34:38.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:38.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d741405f0 con 0x7f6d740b4d60 2024-03-25T19:34:38.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.750+0000 7f6d7a75d640 1 librados: init done 2024-03-25T19:34:38.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.751+0000 7f6d7a75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:38.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.751+0000 7f6d7a75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:38.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.751+0000 7f6d7a75d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:38.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:38.751+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d740627b0 con 0x7f6d740b4d60 2024-03-25T19:34:39.050 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.051+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d68001cd0 con 0x7f6d740b4d60 2024-03-25T19:34:39.050 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.051+0000 7f6d70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.051+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d68003a60 con 0x7f6d740b4d60 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.051+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.051+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.051+0000 7f6d70ff9640 10 monclient: dump: 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:39.051 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.051+0000 7f6d727fc640 4 set_mon_vals no callback set 2024-03-25T19:34:39.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.051+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f6d68003c80 con 0x7f6d740b4d60 2024-03-25T19:34:39.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.052+0000 7f6d73fff640 10 monclient: get_auth_request con 0x7f6d6006dd00 auth_method 0 2024-03-25T19:34:39.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.052+0000 7f6d70ff9640 10 client.4424.objecter ms_handle_connect 0x7f6d6006dd00 2024-03-25T19:34:39.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.058+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f6d680548f0 con 0x7f6d740b4d60 2024-03-25T19:34:39.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.058+0000 7f6d70ff9640 10 client.4424.objecter ms_dispatch 0x7f6d74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:39.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.058+0000 7f6d70ff9640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:39.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.058+0000 7f6d70ff9640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:39.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.059+0000 7f6d70ff9640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T19:34:39.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.059+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d68050400 con 0x7f6d740b4d60 2024-03-25T19:34:39.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.059+0000 7f6d70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:39.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.059+0000 7f6d70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:39.315 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.315+0000 7f6d7a75d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:39.315 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.315+0000 7f6d7a75d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:39.315 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.315+0000 7f6d7a75d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:39.315 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.315+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d74069f40 con 0x7f6d740b4d60 2024-03-25T19:34:39.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.344+0000 7f6d70ff9640 1 -- 172.21.15.143:0/3797852532 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f6d68052330 con 0x7f6d740b4d60 2024-03-25T19:34:39.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.344+0000 7f6d70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:39.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.344+0000 7f6d70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:39.344 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:34:39.344 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:39.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 10 librados: watch_flush enter 2024-03-25T19:34:39.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 10 librados: watch_flush exit 2024-03-25T19:34:39.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T19:34:39.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 10 client.4424.objecter successfully canceled tick 2024-03-25T19:34:39.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 >> v1:172.21.15.143:6800/1292282666 conn(0x7f6d6006dd00 legacy=0x7f6d600701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:39.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 10 monclient: shutdown 2024-03-25T19:34:39.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:39.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 >> v1:172.21.15.143:6790/0 conn(0x7f6d740b4d60 legacy=0x7f6d7413b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:39.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 shutdown_connections 2024-03-25T19:34:39.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 >> 172.21.15.143:0/3797852532 conn(0x7f6d740617a0 msgr2=0x7f6d74061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:39.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 shutdown_connections 2024-03-25T19:34:39.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 1 -- 172.21.15.143:0/3797852532 wait complete. 2024-03-25T19:34:39.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:39.346+0000 7f6d7a75d640 1 librados: shutdown 2024-03-25T19:34:39.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:46.163 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.280+0000 7f98d7167640 10 monclient: get_monmap_and_config 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.280+0000 7f98d7167640 10 monclient: build_initial_monmap 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.280+0000 7f98d7167640 10 monclient: monmap: 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:34:46.282345+0000 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:34:46.282345+0000 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:46.281 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:34:46.282 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:34:46.282 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.281+0000 7f98d7167640 1 Processor -- start 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.281+0000 7f98d7167640 1 -- start start 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.281+0000 7f98d7167640 10 monclient: init 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.281+0000 7f98d7167640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.281+0000 7f98d7167640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.281+0000 7f98d7167640 10 monclient(hunting): picked mon.noname-b con 0x7f98d00a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.281+0000 7f98d7167640 10 monclient(hunting): picked mon.noname-a con 0x7f98d013f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.282+0000 7f98d7167640 10 monclient(hunting): picked mon.noname-c con 0x7f98d01430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.282+0000 7f98d7167640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0142520 con 0x7f98d013f710 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.282+0000 7f98d7167640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00b28a0 con 0x7f98d01430c0 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.282+0000 7f98d7167640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00b2a30 con 0x7f98d00a98f0 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.282+0000 7f98d7167640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:46.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.282+0000 7f98d7167640 10 monclient(hunting): authenticate will time out at 6302.626829s 2024-03-25T19:34:46.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.288+0000 7f98d5704640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f98d01430c0 0x7f98d014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:60254/0 (socket says 172.21.15.143:60254) 2024-03-25T19:34:46.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.288+0000 7f98d5704640 1 -- 172.21.15.143:0/293407453 learned_addr learned my addr 172.21.15.143:0/293407453 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:46.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.304+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726632461 0 0) 0x7f98d00b2a30 con 0x7f98d00a98f0 2024-03-25T19:34:46.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.304+0000 7f98cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.304+0000 7f98cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.304+0000 7f98cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.304+0000 7f98cf7fe640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:34:46.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.304+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c0003520 con 0x7f98d00a98f0 2024-03-25T19:34:46.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.308+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834705135 0 0) 0x7f98d00b28a0 con 0x7f98d01430c0 2024-03-25T19:34:46.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.308+0000 7f98cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.308+0000 7f98cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.308+0000 7f98cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.308+0000 7f98cf7fe640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:34:46.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.308+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d00b2a30 con 0x7f98d01430c0 2024-03-25T19:34:46.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.311+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177957672 0 0) 0x7f98d0142520 con 0x7f98d013f710 2024-03-25T19:34:46.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.311+0000 7f98cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.311+0000 7f98cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.311+0000 7f98cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.311+0000 7f98cf7fe640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:34:46.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.311+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d00b28a0 con 0x7f98d013f710 2024-03-25T19:34:46.320 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3997939168 0 0) 0x7f98c0003520 con 0x7f98d00a98f0 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 >> v1:172.21.15.203:6789/0 conn(0x7f98d01430c0 legacy=0x7f98d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 >> v1:172.21.15.143:6789/0 conn(0x7f98d013f710 legacy=0x7f98d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 1 monclient: found mon.noname-b 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00b44c0 con 0x7f98d00a98f0 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f98c4001690 con 0x7f98d00a98f0 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98cf7fe640 10 monclient: dump: 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:46.321 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98d7167640 5 monclient: authenticate success, global_id 4427 2024-03-25T19:34:46.322 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.319+0000 7f98d7167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.334+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c40018b0 con 0x7f98d00a98f0 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.334+0000 7f98cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.334+0000 7f98cf7fe640 1 -- 172.21.15.143:0/293407453 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f98c4001c40 con 0x7f98d00a98f0 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.334+0000 7f98cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.334+0000 7f98cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.334+0000 7f98cf7fe640 10 monclient: dump: 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:46.335 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:46.336 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:46.336 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:46.336 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:46.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.334+0000 7f98d7167640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:46.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.334+0000 7f98d7167640 4 set_mon_vals no callback set 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.335+0000 7f98d7167640 10 monclient: shutdown 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.335+0000 7f98d7167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.335+0000 7f98d7167640 1 -- 172.21.15.143:0/293407453 >> v1:172.21.15.143:6790/0 conn(0x7f98d00a98f0 legacy=0x7f98d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.335+0000 7f98d7167640 1 -- 172.21.15.143:0/293407453 shutdown_connections 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.335+0000 7f98d7167640 1 -- 172.21.15.143:0/293407453 >> 172.21.15.143:0/293407453 conn(0x7f98d00b7ea0 msgr2=0x7f98d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.335+0000 7f98d7167640 1 -- 172.21.15.143:0/293407453 shutdown_connections 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.335+0000 7f98d7167640 1 -- 172.21.15.143:0/293407453 wait complete. 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.336+0000 7f98d7167640 10 monclient: build_initial_monmap 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.336+0000 7f98d7167640 10 monclient: monmap: 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:34:46.337 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:34:46.338 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:46.338 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:34:46.338 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:46.338 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:34:46.338 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:34:46.338 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:46.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.336+0000 7f98d7167640 1 librados: starting msgr at 2024-03-25T19:34:46.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.336+0000 7f98d7167640 1 librados: starting objecter 2024-03-25T19:34:46.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 1 Processor -- start 2024-03-25T19:34:46.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 1 -- start start 2024-03-25T19:34:46.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 1 librados: setting wanted keys 2024-03-25T19:34:46.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 1 librados: calling monclient init 2024-03-25T19:34:46.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 10 monclient: init 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 10 monclient(hunting): picked mon.noname-a con 0x7f98d00a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 10 monclient(hunting): picked mon.noname-b con 0x7f98d013f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.337+0000 7f98d7167640 10 monclient(hunting): picked mon.noname-c con 0x7f98d01430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.338+0000 7f98d7167640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0135610 con 0x7f98d00a98f0 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.338+0000 7f98d7167640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0137230 con 0x7f98d013f710 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.338+0000 7f98d7167640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d0135cf0 con 0x7f98d01430c0 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.338+0000 7f98d7167640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:46.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.338+0000 7f98d7167640 10 monclient(hunting): authenticate will time out at 6302.682713s 2024-03-25T19:34:46.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.351+0000 7f98cffff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f98d013f710 0x7f98d01dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:60314/0 (socket says 172.21.15.143:60314) 2024-03-25T19:34:46.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.351+0000 7f98cffff640 1 -- 172.21.15.143:0/3751289993 learned_addr learned my addr 172.21.15.143:0/3751289993 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:46.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.356+0000 7f98cd7fa640 10 client.?.objecter ms_handle_connect 0x7f98d00a98f0 2024-03-25T19:34:46.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.356+0000 7f98cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:46.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.357+0000 7f98cd7fa640 10 client.?.objecter ms_handle_connect 0x7f98d013f710 2024-03-25T19:34:46.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.357+0000 7f98cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:46.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.361+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999775064 0 0) 0x7f98d0135610 con 0x7f98d00a98f0 2024-03-25T19:34:46.361 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.361+0000 7f98cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.361+0000 7f98cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.361+0000 7f98cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.361+0000 7f98cd7fa640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:34:46.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.361+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b4004eb0 con 0x7f98d00a98f0 2024-03-25T19:34:46.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.366+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700159038 0 0) 0x7f98d0137230 con 0x7f98d013f710 2024-03-25T19:34:46.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.366+0000 7f98cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.366+0000 7f98cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.366+0000 7f98cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.366+0000 7f98cd7fa640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:34:46.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.366+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d0135610 con 0x7f98d013f710 2024-03-25T19:34:46.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.368+0000 7f98cd7fa640 10 client.?.objecter ms_handle_connect 0x7f98d01430c0 2024-03-25T19:34:46.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.368+0000 7f98cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.377+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2981026389 0 0) 0x7f98d0135610 con 0x7f98d013f710 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.377+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d0137230 con 0x7f98d013f710 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.377+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f98bc0019f0 con 0x7f98d013f710 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.377+0000 7f98cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.377+0000 7f98cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.377+0000 7f98cd7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.378 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.377+0000 7f98cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.378+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954259582 0 0) 0x7f98d0135cf0 con 0x7f98d01430c0 2024-03-25T19:34:46.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.378+0000 7f98cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:34:46.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.378+0000 7f98cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:46.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.378+0000 7f98cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:46.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.378+0000 7f98cd7fa640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:34:46.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.378+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d0135610 con 0x7f98d01430c0 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.385+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1885977273 0 0) 0x7f98b4004eb0 con 0x7f98d00a98f0 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.385+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d0135cf0 con 0x7f98d00a98f0 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.385+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f98c40015d0 con 0x7f98d00a98f0 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.385+0000 7f98cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.385+0000 7f98cd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.385+0000 7f98cd7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.386 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.387 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.387 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:46.387 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:46.387 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:46.387 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:46.387 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:46.389 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.388+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3926039177 0 0) 0x7f98d0135610 con 0x7f98d01430c0 2024-03-25T19:34:46.389 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.388+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98b4004eb0 con 0x7f98d01430c0 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.388+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f98c8001b50 con 0x7f98d01430c0 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.388+0000 7f98cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.388+0000 7f98cd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.388+0000 7f98cd7fa640 10 monclient(hunting): dump: 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:46.390 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:46.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2770821645 0 0) 0x7f98d0135cf0 con 0x7f98d00a98f0 2024-03-25T19:34:46.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 >> v1:172.21.15.143:6790/0 conn(0x7f98d01430c0 legacy=0x7f98d01e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 >> v1:172.21.15.203:6789/0 conn(0x7f98d013f710 legacy=0x7f98d01dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:34:46.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 1 monclient: found mon.a 2024-03-25T19:34:46.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:46.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:46.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d0135e80 con 0x7f98d00a98f0 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 5 monclient: authenticate success, global_id 4440 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 10 monclient: _renew_subs 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d0131660 con 0x7f98d00a98f0 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 10 monclient: _renew_subs 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d00b7960 con 0x7f98d00a98f0 2024-03-25T19:34:46.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.400+0000 7f98d7167640 1 librados: init done 2024-03-25T19:34:46.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.401+0000 7f98d7167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:46.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.401+0000 7f98d7167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:46.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.401+0000 7f98d7167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:46.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.401+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d00627b0 con 0x7f98d00a98f0 2024-03-25T19:34:46.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.705+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c40038a0 con 0x7f98d00a98f0 2024-03-25T19:34:46.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.705+0000 7f98cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:46.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.705+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f98c4003bb0 con 0x7f98d00a98f0 2024-03-25T19:34:46.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.705+0000 7f98cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.705+0000 7f98cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.705+0000 7f98cd7fa640 10 monclient: dump: 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:46.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.705+0000 7f98ceffd640 4 set_mon_vals no callback set 2024-03-25T19:34:46.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.706+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f98c4019b80 con 0x7f98d00a98f0 2024-03-25T19:34:46.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.707+0000 7f98cffff640 10 monclient: get_auth_request con 0x7f98b406e0c0 auth_method 0 2024-03-25T19:34:46.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.707+0000 7f98cd7fa640 10 client.4440.objecter ms_handle_connect 0x7f98b406e0c0 2024-03-25T19:34:46.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.712+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f98c4053640 con 0x7f98d00a98f0 2024-03-25T19:34:46.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.712+0000 7f98cd7fa640 10 client.4440.objecter ms_dispatch 0x7f98d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:46.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.712+0000 7f98cd7fa640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:46.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.712+0000 7f98cd7fa640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:46.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.712+0000 7f98cd7fa640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-25T19:34:46.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.712+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98c404f150 con 0x7f98d00a98f0 2024-03-25T19:34:46.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.712+0000 7f98cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:46.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.712+0000 7f98cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:46.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.969+0000 7f98d7167640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:46.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.969+0000 7f98d7167640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:46.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.969+0000 7f98d7167640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:34:46.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.969+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98d0136010 con 0x7f98d00a98f0 2024-03-25T19:34:46.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.995+0000 7f98cd7fa640 1 -- 172.21.15.143:0/3751289993 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f98d0136010 con 0x7f98d00a98f0 2024-03-25T19:34:46.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.995+0000 7f98cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:46.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.995+0000 7f98cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:46.996 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:34:46.996 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:46.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.997+0000 7f98d7167640 10 librados: watch_flush enter 2024-03-25T19:34:46.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.997+0000 7f98d7167640 10 librados: watch_flush exit 2024-03-25T19:34:46.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.997+0000 7f98d7167640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-25T19:34:46.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.997+0000 7f98d7167640 10 client.4440.objecter successfully canceled tick 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.997+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 >> v1:172.21.15.143:6800/1292282666 conn(0x7f98b406e0c0 legacy=0x7f98b4070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.997+0000 7f98d7167640 10 monclient: shutdown 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.997+0000 7f98d7167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.997+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 >> v1:172.21.15.143:6789/0 conn(0x7f98d00a98f0 legacy=0x7f98d0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.998+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 shutdown_connections 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.998+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 >> 172.21.15.143:0/3751289993 conn(0x7f98d00b7ea0 msgr2=0x7f98d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.998+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 shutdown_connections 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.998+0000 7f98d7167640 1 -- 172.21.15.143:0/3751289993 wait complete. 2024-03-25T19:34:46.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:46.998+0000 7f98d7167640 1 librados: shutdown 2024-03-25T19:34:47.011 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:34:53.600 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:20.165 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.606 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:20.171 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.655 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:20.220 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.662 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:20.198 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.667 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:20.203 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.669 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:20.234 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.679 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:20.244 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.690 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:20.225 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.703 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:20.268 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:34:53.818 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:34:53.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.928+0000 7f406800f640 10 monclient: get_monmap_and_config 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.928+0000 7f406800f640 10 monclient: build_initial_monmap 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.928+0000 7f406800f640 10 monclient: monmap: 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:34:53.928933+0000 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:34:53.928933+0000 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:34:53.928 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:53.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.928+0000 7f406800f640 1 Processor -- start 2024-03-25T19:34:53.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.929+0000 7f406800f640 1 -- start start 2024-03-25T19:34:53.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.929+0000 7f406800f640 10 monclient: init 2024-03-25T19:34:53.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.929+0000 7f406800f640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:53.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.929+0000 7f406800f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:53.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.929+0000 7f406800f640 10 monclient(hunting): picked mon.noname-c con 0x7f40600b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:53.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.929+0000 7f406800f640 10 monclient(hunting): picked mon.noname-b con 0x7f40600b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:53.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.930+0000 7f406800f640 10 monclient(hunting): picked mon.noname-a con 0x7f406014be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:53.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.930+0000 7f406800f640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600af940 con 0x7f406014be90 2024-03-25T19:34:53.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.930+0000 7f406800f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600b0a70 con 0x7f40600b4d60 2024-03-25T19:34:53.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.930+0000 7f406800f640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40600b0d50 con 0x7f40600b1730 2024-03-25T19:34:53.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.930+0000 7f406800f640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:53.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.930+0000 7f406800f640 10 monclient(hunting): authenticate will time out at 6310.273787s 2024-03-25T19:34:53.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.938+0000 7f40665ac640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f406014be90 0x7f406014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:36920/0 (socket says 172.21.15.143:36920) 2024-03-25T19:34:53.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.938+0000 7f40665ac640 1 -- 172.21.15.143:0/270203233 learned_addr learned my addr 172.21.15.143:0/270203233 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:53.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.951+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949619479 0 0) 0x7f40600af940 con 0x7f406014be90 2024-03-25T19:34:53.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.951+0000 7f4064da9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:53.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.951+0000 7f4064da9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:53.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.951+0000 7f4064da9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:53.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.951+0000 7f4064da9640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:34:53.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.951+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c003520 con 0x7f406014be90 2024-03-25T19:34:53.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.957+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962323596 0 0) 0x7f40600b0a70 con 0x7f40600b4d60 2024-03-25T19:34:53.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.957+0000 7f4064da9640 10 monclient(hunting): _negotiate 2024-03-25T19:34:53.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.957+0000 7f4064da9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:53.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.957+0000 7f4064da9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:53.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.957+0000 7f4064da9640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:34:53.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.957+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40600af940 con 0x7f40600b4d60 2024-03-25T19:34:53.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4012901804 0 0) 0x7f404c003520 con 0x7f406014be90 2024-03-25T19:34:53.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 >> v1:172.21.15.143:6790/0 conn(0x7f40600b1730 legacy=0x7f40600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:53.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 >> v1:172.21.15.203:6789/0 conn(0x7f40600b4d60 legacy=0x7f406013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:53.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 10 monclient: _finish_hunting 0 2024-03-25T19:34:53.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 1 monclient: found mon.noname-a 2024-03-25T19:34:53.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:53.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40600b1030 con 0x7f406014be90 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 10 monclient: _finish_auth 0 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 10 monclient: _check_auth_tickets 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4058001a10 con 0x7f406014be90 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f4064da9640 10 monclient: dump: 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:53.960 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:53.961 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:53.961 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:53.961 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:53.961 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:53.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f406800f640 5 monclient: authenticate success, global_id 4443 2024-03-25T19:34:53.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.959+0000 7f406800f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.974+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4058001bf0 con 0x7f406014be90 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.974+0000 7f4064da9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.974+0000 7f4064da9640 1 -- 172.21.15.143:0/270203233 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4058002000 con 0x7f406014be90 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.974+0000 7f4064da9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.974+0000 7f4064da9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.974+0000 7f4064da9640 10 monclient: dump: 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:53.974 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.974+0000 7f406800f640 10 monclient: get_monmap_and_config success 2024-03-25T19:34:53.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.974+0000 7f406800f640 4 set_mon_vals no callback set 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.975+0000 7f406800f640 10 monclient: shutdown 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.975+0000 7f406800f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.975+0000 7f406800f640 1 -- 172.21.15.143:0/270203233 >> v1:172.21.15.143:6789/0 conn(0x7f406014be90 legacy=0x7f406014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.975+0000 7f406800f640 1 -- 172.21.15.143:0/270203233 shutdown_connections 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.975+0000 7f406800f640 1 -- 172.21.15.143:0/270203233 >> 172.21.15.143:0/270203233 conn(0x7f40600617a0 msgr2=0x7f4060061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.975+0000 7f406800f640 1 -- 172.21.15.143:0/270203233 shutdown_connections 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.975+0000 7f406800f640 1 -- 172.21.15.143:0/270203233 wait complete. 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 10 monclient: build_initial_monmap 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 10 monclient: monmap: 2024-03-25T19:34:53.977 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 1 librados: starting msgr at 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 1 librados: starting objecter 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 1 Processor -- start 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 1 -- start start 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 1 librados: setting wanted keys 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 1 librados: calling monclient init 2024-03-25T19:34:53.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.976+0000 7f406800f640 10 monclient: init 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 10 monclient(hunting): picked mon.noname-c con 0x7f40600b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 10 monclient(hunting): picked mon.noname-a con 0x7f40600b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 10 monclient(hunting): picked mon.noname-b con 0x7f406014be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40601371d0 con 0x7f40600b4d60 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4060145220 con 0x7f406014be90 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4060144d90 con 0x7f40600b1730 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:53.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.977+0000 7f406800f640 10 monclient(hunting): authenticate will time out at 6310.321139s 2024-03-25T19:34:53.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.994+0000 7f4065dab640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f40600b1730 0x7f4060135de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:53902/0 (socket says 172.21.15.143:53902) 2024-03-25T19:34:53.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.994+0000 7f4065dab640 1 -- 172.21.15.143:0/1412130068 learned_addr learned my addr 172.21.15.143:0/1412130068 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:34:53.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.995+0000 7f40567fc640 10 client.?.objecter ms_handle_connect 0x7f40600b1730 2024-03-25T19:34:53.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.995+0000 7f40567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:53.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.999+0000 7f40567fc640 10 client.?.objecter ms_handle_connect 0x7f406014be90 2024-03-25T19:34:53.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:53.999+0000 7f40567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:54.000 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.001+0000 7f40567fc640 10 client.?.objecter ms_handle_connect 0x7f40600b4d60 2024-03-25T19:34:54.000 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.001+0000 7f40567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:34:54.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.006+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271724668 0 0) 0x7f4060145220 con 0x7f406014be90 2024-03-25T19:34:54.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.006+0000 7f40567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:54.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.006+0000 7f40567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:54.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.006+0000 7f40567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:54.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.006+0000 7f40567fc640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:34:54.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.006+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4044004eb0 con 0x7f406014be90 2024-03-25T19:34:54.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.010+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018116948 0 0) 0x7f4060144d90 con 0x7f40600b1730 2024-03-25T19:34:54.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.010+0000 7f40567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:54.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.010+0000 7f40567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:54.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.010+0000 7f40567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:54.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.010+0000 7f40567fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:34:54.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.010+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4060145220 con 0x7f40600b1730 2024-03-25T19:34:54.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.016+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3658249481 0 0) 0x7f4060145220 con 0x7f40600b1730 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.016+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4060144d90 con 0x7f40600b1730 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.016+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f405c0019f0 con 0x7f40600b1730 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.016+0000 7f40567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.016+0000 7f40567fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.016+0000 7f40567fc640 10 monclient(hunting): dump: 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:54.017 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:54.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.016+0000 7f40567fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:34:54.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.017+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830689839 0 0) 0x7f40601371d0 con 0x7f40600b4d60 2024-03-25T19:34:54.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.017+0000 7f40567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:54.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.017+0000 7f40567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:54.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.017+0000 7f40567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:34:54.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.017+0000 7f40567fc640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:34:54.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.017+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4060145220 con 0x7f40600b4d60 2024-03-25T19:34:54.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4269835131 0 0) 0x7f4060144d90 con 0x7f40600b1730 2024-03-25T19:34:54.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.203:6789/0 conn(0x7f406014be90 legacy=0x7f40601e25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:54.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6789/0 conn(0x7f40600b4d60 legacy=0x7f4060137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:54.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:54.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 1 monclient: found mon.c 2024-03-25T19:34:54.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:54.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060144f20 con 0x7f40600b1730 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f40567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 5 monclient: authenticate success, global_id 4436 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 10 monclient: _renew_subs 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4060131c10 con 0x7f40600b1730 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 10 monclient: _renew_subs 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4060145500 con 0x7f40600b1730 2024-03-25T19:34:54.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.022+0000 7f406800f640 1 librados: init done 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f406800f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f406800f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f406800f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40601401f0 con 0x7f40600b1730 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f4065dab640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6790/0 conn(0x7f40600b1730 legacy=0x7f4060135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f4065dab640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6790/0 conn(0x7f40600b1730 legacy=0x7f4060135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f4065dab640 1 --1- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6790/0 conn(0x7f40600b1730 0x7f4060135de0 :-1 s=OPENED pgs=34681 cs=1 l=1).handle_message read tag failed 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f4065dab640 1 --1- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6790/0 conn(0x7f40600b1730 0x7f4060135de0 :-1 s=OPENED pgs=34681 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f40567fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.143:6790/0 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f40567fc640 10 monclient: _reopen_session rank -1 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.023+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6790/0 conn(0x7f40600b1730 legacy=0x7f4060135de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:34:54.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.024+0000 7f40567fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:34:54.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.024+0000 7f40567fc640 10 monclient(hunting): picked mon.b con 0x7f406014be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:34:54.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.024+0000 7f40567fc640 10 monclient(hunting): picked mon.c con 0x7f4044006c20 addr v1:172.21.15.143:6790/0 2024-03-25T19:34:54.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.024+0000 7f40567fc640 10 monclient(hunting): picked mon.a con 0x7f40600b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:34:54.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.024+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4060144d90 con 0x7f40600b4d60 2024-03-25T19:34:54.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.024+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f40601371d0 con 0x7f406014be90 2024-03-25T19:34:54.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.024+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4044013a40 con 0x7f4044006c20 2024-03-25T19:34:54.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.024+0000 7f40567fc640 10 monclient(hunting): _renew_subs 2024-03-25T19:34:54.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.040+0000 7f40567fc640 10 client.4436.objecter ms_handle_connect 0x7f406014be90 2024-03-25T19:34:54.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.040+0000 7f40567fc640 10 client.4436.objecter resend_mon_ops 2024-03-25T19:34:54.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.045+0000 7f40567fc640 10 client.4436.objecter ms_handle_connect 0x7f40600b4d60 2024-03-25T19:34:54.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.045+0000 7f40567fc640 10 client.4436.objecter resend_mon_ops 2024-03-25T19:34:54.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.054+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116717538 0 0) 0x7f4060144d90 con 0x7f40600b4d60 2024-03-25T19:34:54.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.054+0000 7f40567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:54.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.054+0000 7f40567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:54.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.054+0000 7f40567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:34:54.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.054+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.143:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f4044011940 con 0x7f40600b4d60 2024-03-25T19:34:54.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.060+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681301169 0 0) 0x7f40601371d0 con 0x7f406014be90 2024-03-25T19:34:54.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.060+0000 7f40567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:34:54.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.060+0000 7f40567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:34:54.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.060+0000 7f40567fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:34:54.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.060+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.203:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f4060144d90 con 0x7f406014be90 2024-03-25T19:34:54.078 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2164475669 0 0) 0x7f4060144d90 con 0x7f406014be90 2024-03-25T19:34:54.078 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6790/0 conn(0x7f4044006c20 legacy=0x7f404400c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6789/0 conn(0x7f40600b4d60 legacy=0x7f404400fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 1 monclient: found mon.b 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f4044014e20 con 0x7f406014be90 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.078+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4044013a40 con 0x7f406014be90 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.079+0000 7f40567fc640 10 monclient: _finish_auth 0 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.079+0000 7f40567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.079+0000 7f40567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.079+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f405c003a10 con 0x7f406014be90 2024-03-25T19:34:54.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.079+0000 7f40567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.079+0000 7f40567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.079+0000 7f40567fc640 10 monclient: dump: 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:34:54.080 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:34:54.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.359+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f405c003b70 con 0x7f406014be90 2024-03-25T19:34:54.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.359+0000 7f40567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:34:54.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.359+0000 7f4057fff640 4 set_mon_vals no callback set 2024-03-25T19:34:54.360 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.360+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f405c003da0 con 0x7f406014be90 2024-03-25T19:34:54.360 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.361+0000 7f40655aa640 10 monclient: get_auth_request con 0x7f40600b1730 auth_method 0 2024-03-25T19:34:54.360 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.361+0000 7f40567fc640 10 client.4436.objecter ms_handle_connect 0x7f40600b1730 2024-03-25T19:34:54.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.365+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f405c0549c0 con 0x7f406014be90 2024-03-25T19:34:54.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.365+0000 7f40567fc640 10 client.4436.objecter ms_dispatch 0x7f4060000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:34:54.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.365+0000 7f40567fc640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:34:54.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.365+0000 7f40567fc640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:34:54.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.365+0000 7f40567fc640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-25T19:34:54.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.365+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f405c050300 con 0x7f406014be90 2024-03-25T19:34:54.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.365+0000 7f40567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:34:54.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.365+0000 7f40567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:34:54.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.618+0000 7f406800f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:54.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.618+0000 7f406800f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:54.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.618+0000 7f406800f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:34:54.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.618+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40600627b0 con 0x7f406014be90 2024-03-25T19:34:54.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.647+0000 7f40567fc640 1 -- 172.21.15.143:0/1412130068 <== mon.1 v1:172.21.15.203:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f405c052400 con 0x7f406014be90 2024-03-25T19:34:54.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.647+0000 7f40567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:34:54.647 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.647+0000 7f40567fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:34:54.647 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:34:54.647 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 10 librados: watch_flush enter 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 10 librados: watch_flush exit 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 10 client.4436.objecter successfully canceled tick 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.143:6800/1292282666 conn(0x7f40600b1730 legacy=0x7f404407bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 10 monclient: shutdown 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 >> v1:172.21.15.203:6789/0 conn(0x7f406014be90 legacy=0x7f4044006440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 shutdown_connections 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 >> 172.21.15.143:0/1412130068 conn(0x7f40600617a0 msgr2=0x7f406013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:34:54.651 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 shutdown_connections 2024-03-25T19:34:54.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 1 -- 172.21.15.143:0/1412130068 wait complete. 2024-03-25T19:34:54.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:34:54.650+0000 7f406800f640 1 librados: shutdown 2024-03-25T19:34:54.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:01.469 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.571+0000 7fb080197640 10 monclient: get_monmap_and_config 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.571+0000 7fb080197640 10 monclient: build_initial_monmap 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.571+0000 7fb080197640 10 monclient: monmap: 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:35:01.572404+0000 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:35:01.572404+0000 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:35:01.571 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:35:01.572 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:01.572 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 1 Processor -- start 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 1 -- start start 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 10 monclient: init 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 10 monclient(hunting): picked mon.noname-c con 0x7fb0780a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 10 monclient(hunting): picked mon.noname-b con 0x7fb07813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 10 monclient(hunting): picked mon.noname-a con 0x7fb0781430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb078142520 con 0x7fb0781430c0 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0780b28a0 con 0x7fb0780a98f0 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0780b2a30 con 0x7fb07813f710 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.572+0000 7fb080197640 10 monclient(hunting): authenticate will time out at 6317.916493s 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.573+0000 7fb07d732640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fb07813f710 0x7fb0781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:38330/0 (socket says 172.21.15.143:38330) 2024-03-25T19:35:01.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.573+0000 7fb07d732640 1 -- 172.21.15.143:0/3424897630 learned_addr learned my addr 172.21.15.143:0/3424897630 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:01.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.605+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020851151 0 0) 0x7fb078142520 con 0x7fb0781430c0 2024-03-25T19:35:01.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.605+0000 7fb07cf31640 10 monclient(hunting): _negotiate 2024-03-25T19:35:01.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.605+0000 7fb07cf31640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:01.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.605+0000 7fb07cf31640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:01.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.605+0000 7fb07cf31640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:35:01.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.605+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb064003520 con 0x7fb0781430c0 2024-03-25T19:35:01.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.608+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636771298 0 0) 0x7fb0780b28a0 con 0x7fb0780a98f0 2024-03-25T19:35:01.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.608+0000 7fb07cf31640 10 monclient(hunting): _negotiate 2024-03-25T19:35:01.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.608+0000 7fb07cf31640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:01.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.608+0000 7fb07cf31640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:01.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.608+0000 7fb07cf31640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:35:01.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.608+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb078142520 con 0x7fb0780a98f0 2024-03-25T19:35:01.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484787875 0 0) 0x7fb078142520 con 0x7fb0780a98f0 2024-03-25T19:35:01.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 >> v1:172.21.15.143:6790/0 conn(0x7fb07813f710 legacy=0x7fb0781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:01.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 >> v1:172.21.15.143:6789/0 conn(0x7fb0781430c0 legacy=0x7fb07814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:01.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 10 monclient: _finish_hunting 0 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 1 monclient: found mon.noname-c 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0780b44c0 con 0x7fb0780a98f0 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 10 monclient: _finish_auth 0 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 10 monclient: _check_auth_tickets 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb080197640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb074001670 con 0x7fb0780a98f0 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb080197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb07cf31640 10 monclient: dump: 2024-03-25T19:35:01.617 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:01.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.616+0000 7fb080197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.623+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0740017d0 con 0x7fb0780a98f0 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.623+0000 7fb07cf31640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.623+0000 7fb07cf31640 1 -- 172.21.15.143:0/3424897630 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb074001ba0 con 0x7fb0780a98f0 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.623+0000 7fb07cf31640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.623+0000 7fb07cf31640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.623+0000 7fb07cf31640 10 monclient: dump: 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:01.623 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:01.624 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:01.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.623+0000 7fb080197640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:01.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.623+0000 7fb080197640 4 set_mon_vals no callback set 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.624+0000 7fb080197640 10 monclient: shutdown 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.624+0000 7fb080197640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.624+0000 7fb080197640 1 -- 172.21.15.143:0/3424897630 >> v1:172.21.15.203:6789/0 conn(0x7fb0780a98f0 legacy=0x7fb0780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.624+0000 7fb080197640 1 -- 172.21.15.143:0/3424897630 shutdown_connections 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.624+0000 7fb080197640 1 -- 172.21.15.143:0/3424897630 >> 172.21.15.143:0/3424897630 conn(0x7fb0780b7ea0 msgr2=0x7fb0780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.624+0000 7fb080197640 1 -- 172.21.15.143:0/3424897630 shutdown_connections 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.624+0000 7fb080197640 1 -- 172.21.15.143:0/3424897630 wait complete. 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 10 monclient: build_initial_monmap 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 10 monclient: monmap: 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:01.625 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 1 librados: starting msgr at 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 1 librados: starting objecter 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 1 Processor -- start 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 1 -- start start 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 1 librados: setting wanted keys 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 1 librados: calling monclient init 2024-03-25T19:35:01.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.625+0000 7fb080197640 10 monclient: init 2024-03-25T19:35:01.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:01.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:01.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 10 monclient(hunting): picked mon.noname-b con 0x7fb0780a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:01.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 10 monclient(hunting): picked mon.noname-c con 0x7fb07813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:01.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 10 monclient(hunting): picked mon.noname-a con 0x7fb0781430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:01.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07813b3d0 con 0x7fb0781430c0 2024-03-25T19:35:01.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07813b6b0 con 0x7fb0780a98f0 2024-03-25T19:35:01.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0780b7b70 con 0x7fb07813f710 2024-03-25T19:35:01.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:01.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.626+0000 7fb080197640 10 monclient(hunting): authenticate will time out at 6317.969915s 2024-03-25T19:35:01.636 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.636+0000 7fb07df33640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb0780a98f0 0x7fb0780b7460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:52794/0 (socket says 172.21.15.143:52794) 2024-03-25T19:35:01.636 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.636+0000 7fb07df33640 1 -- 172.21.15.143:0/1337197677 learned_addr learned my addr 172.21.15.143:0/1337197677 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:01.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.638+0000 7fb06e7fc640 10 client.?.objecter ms_handle_connect 0x7fb0780a98f0 2024-03-25T19:35:01.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.638+0000 7fb06e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:01.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.656+0000 7fb06e7fc640 10 client.?.objecter ms_handle_connect 0x7fb0781430c0 2024-03-25T19:35:01.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.656+0000 7fb06e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:01.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563890867 0 0) 0x7fb07813b6b0 con 0x7fb0780a98f0 2024-03-25T19:35:01.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:01.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:01.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:01.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:35:01.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05c004eb0 con 0x7fb0780a98f0 2024-03-25T19:35:01.670 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838279986 0 0) 0x7fb07813b3d0 con 0x7fb0781430c0 2024-03-25T19:35:01.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:01.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:01.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:01.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:35:01.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.670+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07813b6b0 con 0x7fb0781430c0 2024-03-25T19:35:01.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.678+0000 7fb06e7fc640 10 client.?.objecter ms_handle_connect 0x7fb07813f710 2024-03-25T19:35:01.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.678+0000 7fb06e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:01.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.682+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2421085301 0 0) 0x7fb05c004eb0 con 0x7fb0780a98f0 2024-03-25T19:35:01.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.682+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb07813b3d0 con 0x7fb0780a98f0 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.682+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb0740015f0 con 0x7fb0780a98f0 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.682+0000 7fb06e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.682+0000 7fb06e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.682+0000 7fb06e7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:01.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.682+0000 7fb06e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:01.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.684+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952139225 0 0) 0x7fb0780b7b70 con 0x7fb07813f710 2024-03-25T19:35:01.684 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.684+0000 7fb06e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:01.684 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.684+0000 7fb06e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:01.684 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.684+0000 7fb06e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:01.684 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.684+0000 7fb06e7fc640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:35:01.684 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.684+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05c004eb0 con 0x7fb07813f710 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.686+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 980511090 0 0) 0x7fb07813b6b0 con 0x7fb0781430c0 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.686+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0780b7b70 con 0x7fb0781430c0 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.686+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb070001b90 con 0x7fb0781430c0 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.686+0000 7fb06e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.686+0000 7fb06e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.686+0000 7fb06e7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:01.686 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:01.687 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:01.687 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:01.687 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:01.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1035805029 0 0) 0x7fb07813b3d0 con 0x7fb0780a98f0 2024-03-25T19:35:01.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 >> v1:172.21.15.143:6790/0 conn(0x7fb07813f710 legacy=0x7fb0781df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:01.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 >> v1:172.21.15.143:6789/0 conn(0x7fb0781430c0 legacy=0x7fb0781e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:01.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 1 monclient: found mon.b 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0780b6a80 con 0x7fb0780a98f0 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb06e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb080197640 5 monclient: authenticate success, global_id 4453 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb080197640 10 monclient: _renew_subs 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb080197640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.693+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07813a830 con 0x7fb0780a98f0 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.694+0000 7fb080197640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:01.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.694+0000 7fb080197640 10 monclient: _renew_subs 2024-03-25T19:35:01.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.694+0000 7fb080197640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:35:01.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.694+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07813bdb0 con 0x7fb0780a98f0 2024-03-25T19:35:01.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.694+0000 7fb080197640 1 librados: init done 2024-03-25T19:35:01.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.695+0000 7fb080197640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:01.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.695+0000 7fb080197640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:01.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.695+0000 7fb080197640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:35:01.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.695+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb07813c0b0 con 0x7fb0780a98f0 2024-03-25T19:35:01.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.978+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0740038a0 con 0x7fb0780a98f0 2024-03-25T19:35:01.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.978+0000 7fb06e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:01.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.978+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb074003b10 con 0x7fb0780a98f0 2024-03-25T19:35:01.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.978+0000 7fb06e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:01.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.978+0000 7fb06e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:01.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.978+0000 7fb06e7fc640 10 monclient: dump: 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:01.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.978+0000 7fb06ffff640 4 set_mon_vals no callback set 2024-03-25T19:35:01.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.979+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fb074003d30 con 0x7fb0780a98f0 2024-03-25T19:35:01.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.980+0000 7fb07d732640 10 monclient: get_auth_request con 0x7fb05c06dfb0 auth_method 0 2024-03-25T19:35:01.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.980+0000 7fb06e7fc640 10 client.4453.objecter ms_handle_connect 0x7fb05c06dfb0 2024-03-25T19:35:01.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.985+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fb074054800 con 0x7fb0780a98f0 2024-03-25T19:35:01.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.985+0000 7fb06e7fc640 10 client.4453.objecter ms_dispatch 0x7fb078000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:01.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.985+0000 7fb06e7fc640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:01.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.985+0000 7fb06e7fc640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:01.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.985+0000 7fb06e7fc640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-25T19:35:01.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.985+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb074050310 con 0x7fb0780a98f0 2024-03-25T19:35:01.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.985+0000 7fb06e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:01.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:01.985+0000 7fb06e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:02.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.232+0000 7fb080197640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:02.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.232+0000 7fb080197640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:02.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.232+0000 7fb080197640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:35:02.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.232+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0780627b0 con 0x7fb0780a98f0 2024-03-25T19:35:02.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.257+0000 7fb06e7fc640 1 -- 172.21.15.143:0/1337197677 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fb074052240 con 0x7fb0780a98f0 2024-03-25T19:35:02.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.257+0000 7fb06e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:02.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.257+0000 7fb06e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:02.258 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:35:02.258 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 10 librados: watch_flush enter 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 10 librados: watch_flush exit 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 10 client.4453.objecter successfully canceled tick 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 >> v1:172.21.15.143:6800/1292282666 conn(0x7fb05c06dfb0 legacy=0x7fb05c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 10 monclient: shutdown 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 >> v1:172.21.15.203:6789/0 conn(0x7fb0780a98f0 legacy=0x7fb0780b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 shutdown_connections 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.260+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 >> 172.21.15.143:0/1337197677 conn(0x7fb0780b7ea0 msgr2=0x7fb078074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.261+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 shutdown_connections 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.261+0000 7fb080197640 1 -- 172.21.15.143:0/1337197677 wait complete. 2024-03-25T19:35:02.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:02.261+0000 7fb080197640 1 librados: shutdown 2024-03-25T19:35:02.271 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:09.077 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:09.195 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.195+0000 7f389aa00640 10 monclient: get_monmap_and_config 2024-03-25T19:35:09.195 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.195+0000 7f389aa00640 10 monclient: build_initial_monmap 2024-03-25T19:35:09.195 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.195+0000 7f389aa00640 10 monclient: monmap: 2024-03-25T19:35:09.195 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:09.195 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:09.195 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:35:09.196539+0000 2024-03-25T19:35:09.195 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:35:09.196539+0000 2024-03-25T19:35:09.195 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:09.196 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:09.196 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:09.196 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:35:09.196 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:35:09.196 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.196+0000 7f389aa00640 1 Processor -- start 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.196+0000 7f389aa00640 1 -- start start 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.196+0000 7f389aa00640 10 monclient: init 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 10 monclient(hunting): picked mon.noname-c con 0x7f389406c380 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 10 monclient(hunting): picked mon.noname-a con 0x7f38940b12d0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 10 monclient(hunting): picked mon.noname-b con 0x7f38940aedf0 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38940b16f0 con 0x7f38940b12d0 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38940adfe0 con 0x7f389406c380 2024-03-25T19:35:09.197 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38940add00 con 0x7f38940aedf0 2024-03-25T19:35:09.198 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:09.198 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.197+0000 7f389aa00640 10 monclient(hunting): authenticate will time out at 6325.541272s 2024-03-25T19:35:09.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.208+0000 7f3898f9d640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f38940aedf0 0x7f389414e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:58320/0 (socket says 172.21.15.143:58320) 2024-03-25T19:35:09.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.208+0000 7f3898f9d640 1 -- 172.21.15.143:0/3463365790 learned_addr learned my addr 172.21.15.143:0/3463365790 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.214+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853490936 0 0) 0x7f38940b16f0 con 0x7f38940b12d0 2024-03-25T19:35:09.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.214+0000 7f3892ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:09.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.214+0000 7f3892ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:09.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.214+0000 7f3892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:09.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.214+0000 7f3892ffd640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:35:09.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.214+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f387c003520 con 0x7f38940b12d0 2024-03-25T19:35:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.216+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908240149 0 0) 0x7f38940add00 con 0x7f38940aedf0 2024-03-25T19:35:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.216+0000 7f3892ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.216+0000 7f3892ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.216+0000 7f3892ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.216+0000 7f3892ffd640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:35:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.216+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38940b16f0 con 0x7f38940aedf0 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.228+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 723498065 0 0) 0x7f387c003520 con 0x7f38940b12d0 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 >> v1:172.21.15.143:6790/0 conn(0x7f38940aedf0 legacy=0x7f389414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 >> v1:172.21.15.203:6789/0 conn(0x7f389406c380 legacy=0x7f38940b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 1 monclient: found mon.noname-a 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38940ae2c0 con 0x7f38940b12d0 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:09.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3888001690 con 0x7f38940b12d0 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f389aa00640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f389aa00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f3892ffd640 10 monclient: dump: 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:09.230 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:09.231 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:09.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.229+0000 7f389aa00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.238+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38880017f0 con 0x7f38940b12d0 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.238+0000 7f3892ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.238+0000 7f3892ffd640 1 -- 172.21.15.143:0/3463365790 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3888001b80 con 0x7f38940b12d0 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.238+0000 7f3892ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.238+0000 7f3892ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.238+0000 7f3892ffd640 10 monclient: dump: 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.238 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:09.239 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:09.239 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:09.239 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:09.239 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:09.239 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:09.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.238+0000 7f389aa00640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:09.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.238+0000 7f389aa00640 4 set_mon_vals no callback set 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.239+0000 7f389aa00640 10 monclient: shutdown 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.239+0000 7f389aa00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.239+0000 7f389aa00640 1 -- 172.21.15.143:0/3463365790 >> v1:172.21.15.143:6789/0 conn(0x7f38940b12d0 legacy=0x7f38940ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.239+0000 7f389aa00640 1 -- 172.21.15.143:0/3463365790 shutdown_connections 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.239+0000 7f389aa00640 1 -- 172.21.15.143:0/3463365790 >> 172.21.15.143:0/3463365790 conn(0x7f38940b5d10 msgr2=0x7f38940b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.239+0000 7f389aa00640 1 -- 172.21.15.143:0/3463365790 shutdown_connections 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.239+0000 7f389aa00640 1 -- 172.21.15.143:0/3463365790 wait complete. 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.240+0000 7f389aa00640 10 monclient: build_initial_monmap 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.240+0000 7f389aa00640 10 monclient: monmap: 2024-03-25T19:35:09.240 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.240+0000 7f389aa00640 1 librados: starting msgr at 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.240+0000 7f389aa00640 1 librados: starting objecter 2024-03-25T19:35:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.240+0000 7f389aa00640 1 Processor -- start 2024-03-25T19:35:09.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.241+0000 7f389aa00640 1 -- start start 2024-03-25T19:35:09.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.241+0000 7f389aa00640 1 librados: setting wanted keys 2024-03-25T19:35:09.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.241+0000 7f389aa00640 1 librados: calling monclient init 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.241+0000 7f389aa00640 10 monclient: init 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.241+0000 7f389aa00640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.241+0000 7f389aa00640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.241+0000 7f389aa00640 10 monclient(hunting): picked mon.noname-b con 0x7f389406c380 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.241+0000 7f389aa00640 10 monclient(hunting): picked mon.noname-a con 0x7f38940aedf0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.242+0000 7f389aa00640 10 monclient(hunting): picked mon.noname-c con 0x7f38940b12d0 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.242+0000 7f389aa00640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389406c8e0 con 0x7f38940aedf0 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.242+0000 7f389aa00640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38941c5370 con 0x7f389406c380 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.242+0000 7f389aa00640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38941dedd0 con 0x7f38940b12d0 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.242+0000 7f389aa00640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:09.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.242+0000 7f389aa00640 10 monclient(hunting): authenticate will time out at 6325.585830s 2024-03-25T19:35:09.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.253+0000 7f38937fe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f38940aedf0 0x7f38941df290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:33568/0 (socket says 172.21.15.143:33568) 2024-03-25T19:35:09.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.253+0000 7f38937fe640 1 -- 172.21.15.143:0/2932152326 learned_addr learned my addr 172.21.15.143:0/2932152326 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:09.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.258+0000 7f3890ff9640 10 client.?.objecter ms_handle_connect 0x7f38940aedf0 2024-03-25T19:35:09.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.258+0000 7f3890ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:09.268 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.269+0000 7f3890ff9640 10 client.?.objecter ms_handle_connect 0x7f389406c380 2024-03-25T19:35:09.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.269+0000 7f3890ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:09.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.269+0000 7f3890ff9640 10 client.?.objecter ms_handle_connect 0x7f38940b12d0 2024-03-25T19:35:09.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.269+0000 7f3890ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:09.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.274+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965032500 0 0) 0x7f389406c8e0 con 0x7f38940aedf0 2024-03-25T19:35:09.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.274+0000 7f3890ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:09.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.274+0000 7f3890ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:09.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.274+0000 7f3890ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:09.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.274+0000 7f3890ff9640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:35:09.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.274+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3880004eb0 con 0x7f38940aedf0 2024-03-25T19:35:09.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.280+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312046846 0 0) 0x7f38941dedd0 con 0x7f38940b12d0 2024-03-25T19:35:09.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.280+0000 7f3890ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:09.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.280+0000 7f3890ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:09.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.280+0000 7f3890ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:09.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.280+0000 7f3890ff9640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:35:09.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.280+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389406c8e0 con 0x7f38940b12d0 2024-03-25T19:35:09.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.282+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551620533 0 0) 0x7f38940adfe0 con 0x7f389406c380 2024-03-25T19:35:09.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.282+0000 7f3890ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:09.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.282+0000 7f3890ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:09.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.282+0000 7f3890ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:09.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.282+0000 7f3890ff9640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:35:09.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.282+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38941dedd0 con 0x7f389406c380 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.304+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3755282674 0 0) 0x7f3880004eb0 con 0x7f38940aedf0 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.304+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38940adfe0 con 0x7f38940aedf0 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.304+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3888001b40 con 0x7f38940aedf0 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.304+0000 7f3890ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.304+0000 7f3890ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.304+0000 7f3890ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:09.304 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:09.305 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:09.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.304+0000 7f3890ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:09.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.305+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 267703576 0 0) 0x7f38941dedd0 con 0x7f389406c380 2024-03-25T19:35:09.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.305+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3880004eb0 con 0x7f389406c380 2024-03-25T19:35:09.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.305+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f38840016d0 con 0x7f389406c380 2024-03-25T19:35:09.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.305+0000 7f3890ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.305+0000 7f3890ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.305+0000 7f3890ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:09.306 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.321+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 465363867 0 0) 0x7f389406c8e0 con 0x7f38940b12d0 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.321+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38941dedd0 con 0x7f38940b12d0 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.321+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f388c001c70 con 0x7f38940b12d0 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.321+0000 7f3890ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.321+0000 7f3890ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.321+0000 7f3890ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:09.321 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.322 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.322 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:09.322 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:09.322 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:09.322 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:09.322 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:09.322 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:09.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3839962999 0 0) 0x7f38940adfe0 con 0x7f38940aedf0 2024-03-25T19:35:09.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 >> v1:172.21.15.143:6790/0 conn(0x7f38940b12d0 legacy=0x7f38940acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:09.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 >> v1:172.21.15.203:6789/0 conn(0x7f389406c380 legacy=0x7f38940aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:09.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:09.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 1 monclient: found mon.a 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38941c7490 con 0x7f38940aedf0 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f3890ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 5 monclient: authenticate success, global_id 4458 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 10 monclient: _renew_subs 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38940aafe0 con 0x7f38940aedf0 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 10 monclient: _renew_subs 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38941df9a0 con 0x7f38940aedf0 2024-03-25T19:35:09.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.328+0000 7f389aa00640 1 librados: init done 2024-03-25T19:35:09.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.329+0000 7f389aa00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:09.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.329+0000 7f389aa00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:09.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.329+0000 7f389aa00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:09.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.329+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38940627b0 con 0x7f38940aedf0 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.671+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3888001210 con 0x7f38940aedf0 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.671+0000 7f3890ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.671+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3888001d80 con 0x7f38940aedf0 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.671+0000 7f3890ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.671+0000 7f3890ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.671+0000 7f3890ff9640 10 monclient: dump: 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:09.671 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.671+0000 7f38927fc640 4 set_mon_vals no callback set 2024-03-25T19:35:09.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.672+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f3888003b00 con 0x7f38940aedf0 2024-03-25T19:35:09.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.672+0000 7f3893fff640 10 monclient: get_auth_request con 0x7f388006e0f0 auth_method 0 2024-03-25T19:35:09.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.672+0000 7f3890ff9640 10 client.4458.objecter ms_handle_connect 0x7f388006e0f0 2024-03-25T19:35:09.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.676+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f38880547a0 con 0x7f38940aedf0 2024-03-25T19:35:09.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.676+0000 7f3890ff9640 10 client.4458.objecter ms_dispatch 0x7f3894000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:09.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.676+0000 7f3890ff9640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:09.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.676+0000 7f3890ff9640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:09.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.676+0000 7f3890ff9640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-25T19:35:09.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.676+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38880502b0 con 0x7f38940aedf0 2024-03-25T19:35:09.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.676+0000 7f3890ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:09.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.676+0000 7f3890ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:09.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.907+0000 7f389aa00640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:09.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.907+0000 7f389aa00640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:09.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.907+0000 7f389aa00640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:09.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.907+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3894069f40 con 0x7f38940aedf0 2024-03-25T19:35:09.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.942+0000 7f3890ff9640 1 -- 172.21.15.143:0/2932152326 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f38880521e0 con 0x7f38940aedf0 2024-03-25T19:35:09.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.942+0000 7f3890ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:09.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.942+0000 7f3890ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:09.942 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:35:09.942 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:09.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 10 librados: watch_flush enter 2024-03-25T19:35:09.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 10 librados: watch_flush exit 2024-03-25T19:35:09.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-25T19:35:09.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 10 client.4458.objecter successfully canceled tick 2024-03-25T19:35:09.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 >> v1:172.21.15.143:6800/1292282666 conn(0x7f388006e0f0 legacy=0x7f38800705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:09.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 10 monclient: shutdown 2024-03-25T19:35:09.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:09.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 >> v1:172.21.15.143:6789/0 conn(0x7f38940aedf0 legacy=0x7f38941df290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:09.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 shutdown_connections 2024-03-25T19:35:09.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 >> 172.21.15.143:0/2932152326 conn(0x7f38940b5d10 msgr2=0x7f38941cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:09.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 shutdown_connections 2024-03-25T19:35:09.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 1 -- 172.21.15.143:0/2932152326 wait complete. 2024-03-25T19:35:09.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:09.945+0000 7f389aa00640 1 librados: shutdown 2024-03-25T19:35:09.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:13.622 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:40.187 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:13.676 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:40.241 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:13.709 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:05:40.274 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:13.815 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:40.270 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:13.815 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:40.277 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:13.816 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:05:40.303 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:13.816 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:40.320 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:13.816 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:40.330 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:13.816 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:05:40.354 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:16.760 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:16.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.868+0000 7f216e3c8640 10 monclient: get_monmap_and_config 2024-03-25T19:35:16.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.868+0000 7f216e3c8640 10 monclient: build_initial_monmap 2024-03-25T19:35:16.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.868+0000 7f216e3c8640 10 monclient: monmap: 2024-03-25T19:35:16.868 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:16.868 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:16.868 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:35:16.869613+0000 2024-03-25T19:35:16.868 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:35:16.869613+0000 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.869+0000 7f216e3c8640 1 Processor -- start 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.869+0000 7f216e3c8640 1 -- start start 2024-03-25T19:35:16.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.869+0000 7f216e3c8640 10 monclient: init 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 10 monclient(hunting): picked mon.noname-b con 0x7f21680b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 10 monclient(hunting): picked mon.noname-c con 0x7f21680b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 10 monclient(hunting): picked mon.noname-a con 0x7f216814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680af940 con 0x7f216814be90 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680b0a70 con 0x7f21680b1730 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680b0d50 con 0x7f21680b4d60 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.870+0000 7f216e3c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:16.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.871+0000 7f216e3c8640 10 monclient(hunting): authenticate will time out at 6333.214653s 2024-03-25T19:35:16.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.874+0000 7f216c965640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f216814be90 0x7f216814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:56048/0 (socket says 172.21.15.143:56048) 2024-03-25T19:35:16.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.874+0000 7f216c965640 1 -- 172.21.15.143:0/93216721 learned_addr learned my addr 172.21.15.143:0/93216721 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:16.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.890+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303380100 0 0) 0x7f21680b0d50 con 0x7f21680b4d60 2024-03-25T19:35:16.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.890+0000 7f2166ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:16.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.890+0000 7f2166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:16.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.890+0000 7f2166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:16.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.890+0000 7f2166ffd640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:35:16.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.890+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2150003520 con 0x7f21680b4d60 2024-03-25T19:35:16.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.892+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375429866 0 0) 0x7f21680af940 con 0x7f216814be90 2024-03-25T19:35:16.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.892+0000 7f2166ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:16.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.892+0000 7f2166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:16.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.892+0000 7f2166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:16.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.892+0000 7f2166ffd640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:35:16.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.892+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21680b0d50 con 0x7f216814be90 2024-03-25T19:35:16.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.898+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644553827 0 0) 0x7f21680b0a70 con 0x7f21680b1730 2024-03-25T19:35:16.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.898+0000 7f2166ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:16.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.898+0000 7f2166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:16.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.898+0000 7f2166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:16.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.898+0000 7f2166ffd640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:35:16.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.898+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21680af940 con 0x7f21680b1730 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.906+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1170992259 0 0) 0x7f21680b0d50 con 0x7f216814be90 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 >> v1:172.21.15.143:6790/0 conn(0x7f21680b4d60 legacy=0x7f216813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 >> v1:172.21.15.203:6789/0 conn(0x7f21680b1730 legacy=0x7f21680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 1 monclient: found mon.noname-a 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680b1030 con 0x7f216814be90 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f21600019f0 con 0x7f216814be90 2024-03-25T19:35:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f2166ffd640 10 monclient: dump: 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f216e3c8640 5 monclient: authenticate success, global_id 4461 2024-03-25T19:35:16.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.907+0000 7f216e3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.910+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2160001b50 con 0x7f216814be90 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.910+0000 7f2166ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.910+0000 7f2166ffd640 1 -- 172.21.15.143:0/93216721 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2160001f60 con 0x7f216814be90 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.910+0000 7f216e3c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.910+0000 7f216e3c8640 4 set_mon_vals no callback set 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.910+0000 7f2166ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.910+0000 7f2166ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.910+0000 7f2166ffd640 10 monclient: dump: 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:16.911 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.911+0000 7f216e3c8640 10 monclient: shutdown 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.911+0000 7f216e3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.911+0000 7f216e3c8640 1 -- 172.21.15.143:0/93216721 >> v1:172.21.15.143:6789/0 conn(0x7f216814be90 legacy=0x7f216814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.911+0000 7f216e3c8640 1 -- 172.21.15.143:0/93216721 shutdown_connections 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.911+0000 7f216e3c8640 1 -- 172.21.15.143:0/93216721 >> 172.21.15.143:0/93216721 conn(0x7f21680617a0 msgr2=0x7f2168061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.911+0000 7f216e3c8640 1 -- 172.21.15.143:0/93216721 shutdown_connections 2024-03-25T19:35:16.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.911+0000 7f216e3c8640 1 -- 172.21.15.143:0/93216721 wait complete. 2024-03-25T19:35:16.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 10 monclient: build_initial_monmap 2024-03-25T19:35:16.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 10 monclient: monmap: 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 1 librados: starting msgr at 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 1 librados: starting objecter 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 1 Processor -- start 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 1 -- start start 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 1 librados: setting wanted keys 2024-03-25T19:35:16.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 1 librados: calling monclient init 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.912+0000 7f216e3c8640 10 monclient: init 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 10 monclient(hunting): picked mon.noname-b con 0x7f21680b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 10 monclient(hunting): picked mon.noname-c con 0x7f21680b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 10 monclient(hunting): picked mon.noname-a con 0x7f216814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21681d77a0 con 0x7f216814be90 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21681d8620 con 0x7f21680b1730 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680aab70 con 0x7f21680b4d60 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.913+0000 7f216e3c8640 10 monclient(hunting): authenticate will time out at 6333.257117s 2024-03-25T19:35:16.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.927+0000 7f2167fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f21680b1730 0x7f21681d7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:59668/0 (socket says 172.21.15.143:59668) 2024-03-25T19:35:16.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.927+0000 7f2167fff640 1 -- 172.21.15.143:0/1465228817 learned_addr learned my addr 172.21.15.143:0/1465228817 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:16.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.927+0000 7f2164ff9640 10 client.?.objecter ms_handle_connect 0x7f21680b1730 2024-03-25T19:35:16.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.927+0000 7f2164ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:16.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.930+0000 7f2164ff9640 10 client.?.objecter ms_handle_connect 0x7f21680b4d60 2024-03-25T19:35:16.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.930+0000 7f2164ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:16.932 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.932+0000 7f2164ff9640 10 client.?.objecter ms_handle_connect 0x7f216814be90 2024-03-25T19:35:16.932 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.932+0000 7f2164ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301567228 0 0) 0x7f21681d8620 con 0x7f21680b1730 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2154004eb0 con 0x7f21680b1730 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406425956 0 0) 0x7f21681d77a0 con 0x7f216814be90 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:16.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:16.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:16.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:35:16.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.944+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21681d8620 con 0x7f216814be90 2024-03-25T19:35:16.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.949+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258436654 0 0) 0x7f21680aab70 con 0x7f21680b4d60 2024-03-25T19:35:16.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.949+0000 7f2164ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:35:16.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.949+0000 7f2164ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:16.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.949+0000 7f2164ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:16.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.949+0000 7f2164ff9640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:35:16.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.949+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21681d77a0 con 0x7f21680b4d60 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.958+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 834812279 0 0) 0x7f21681d8620 con 0x7f216814be90 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.958+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21680aab70 con 0x7f216814be90 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.958+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2160001e60 con 0x7f216814be90 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.958+0000 7f2164ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.958+0000 7f2164ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.958+0000 7f2164ff9640 10 monclient(hunting): dump: 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:16.958 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:16.959 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:16.959 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:16.959 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:16.959 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:16.959 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:16.959 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:16.959 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.958+0000 7f2164ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:16.968 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3345526699 0 0) 0x7f21680aab70 con 0x7f216814be90 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 >> v1:172.21.15.143:6790/0 conn(0x7f21680b4d60 legacy=0x7f2168142be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 >> v1:172.21.15.203:6789/0 conn(0x7f21680b1730 legacy=0x7f21681d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 1 monclient: found mon.a 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21681432f0 con 0x7f216814be90 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 10 monclient: _finish_auth 0 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.968+0000 7f2164ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 5 monclient: authenticate success, global_id 4464 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 10 monclient: _renew_subs 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21681435d0 con 0x7f216814be90 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:16.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 10 monclient: _renew_subs 2024-03-25T19:35:16.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:16.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2168140560 con 0x7f216814be90 2024-03-25T19:35:16.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.969+0000 7f216e3c8640 1 librados: init done 2024-03-25T19:35:16.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.970+0000 7f216e3c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:16.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.970+0000 7f216e3c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:16.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.970+0000 7f216e3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:16.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:16.970+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21680627b0 con 0x7f216814be90 2024-03-25T19:35:17.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.285+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2160002060 con 0x7f216814be90 2024-03-25T19:35:17.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.285+0000 7f2164ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:17.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.285+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2160004fb0 con 0x7f216814be90 2024-03-25T19:35:17.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.285+0000 7f2164ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:17.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.285+0000 7f2164ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:17.285 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.285+0000 7f2164ff9640 10 monclient: dump: 2024-03-25T19:35:17.285 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:17.285 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:17.286 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.285+0000 7f21667fc640 4 set_mon_vals no callback set 2024-03-25T19:35:17.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.286+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f216001af80 con 0x7f216814be90 2024-03-25T19:35:17.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.287+0000 7f2167fff640 10 monclient: get_auth_request con 0x7f215406ddc0 auth_method 0 2024-03-25T19:35:17.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.287+0000 7f2164ff9640 10 client.4464.objecter ms_handle_connect 0x7f215406ddc0 2024-03-25T19:35:17.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.292+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f2160054ab0 con 0x7f216814be90 2024-03-25T19:35:17.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.292+0000 7f2164ff9640 10 client.4464.objecter ms_dispatch 0x7f2168000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:17.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.292+0000 7f2164ff9640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:17.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.292+0000 7f2164ff9640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:17.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.292+0000 7f2164ff9640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-25T19:35:17.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.292+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21600503f0 con 0x7f216814be90 2024-03-25T19:35:17.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.292+0000 7f2164ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:17.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.292+0000 7f2164ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:17.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.554+0000 7f216e3c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:17.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.554+0000 7f216e3c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:17.554 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.554+0000 7f216e3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:17.554 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.554+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f216814bc70 con 0x7f216814be90 2024-03-25T19:35:17.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.582+0000 7f2164ff9640 1 -- 172.21.15.143:0/1465228817 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f21600524f0 con 0x7f216814be90 2024-03-25T19:35:17.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.582+0000 7f2164ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:17.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.582+0000 7f2164ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:17.582 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:35:17.582 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:17.585 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 10 librados: watch_flush enter 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 10 librados: watch_flush exit 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 10 client.4464.objecter successfully canceled tick 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 >> v1:172.21.15.143:6800/1292282666 conn(0x7f215406ddc0 legacy=0x7f2154070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 10 monclient: shutdown 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 >> v1:172.21.15.143:6789/0 conn(0x7f216814be90 legacy=0x7f21681e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 shutdown_connections 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.585+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 >> 172.21.15.143:0/1465228817 conn(0x7f21680617a0 msgr2=0x7f2168145070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.586+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 shutdown_connections 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.586+0000 7f216e3c8640 1 -- 172.21.15.143:0/1465228817 wait complete. 2024-03-25T19:35:17.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:17.586+0000 7f216e3c8640 1 librados: shutdown 2024-03-25T19:35:17.598 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:24.405 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.521+0000 7fde0ff25640 10 monclient: get_monmap_and_config 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.521+0000 7fde0ff25640 10 monclient: build_initial_monmap 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.521+0000 7fde0ff25640 10 monclient: monmap: 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:35:24.523147+0000 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:35:24.523147+0000 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:35:24.522 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:24.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.522+0000 7fde0ff25640 1 Processor -- start 2024-03-25T19:35:24.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.522+0000 7fde0ff25640 1 -- start start 2024-03-25T19:35:24.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.522+0000 7fde0ff25640 10 monclient: init 2024-03-25T19:35:24.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:24.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:24.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 10 monclient(hunting): picked mon.noname-c con 0x7fde080b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:24.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 10 monclient(hunting): picked mon.noname-b con 0x7fde080b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:24.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 10 monclient(hunting): picked mon.noname-a con 0x7fde0814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:24.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde080af940 con 0x7fde0814be90 2024-03-25T19:35:24.524 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde080b0a70 con 0x7fde080b4d60 2024-03-25T19:35:24.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde080b0d50 con 0x7fde080b1730 2024-03-25T19:35:24.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:24.525 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.523+0000 7fde0ff25640 10 monclient(hunting): authenticate will time out at 6340.868445s 2024-03-25T19:35:24.535 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.534+0000 7fde0dcc1640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fde080b4d60 0x7fde0813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:36446/0 (socket says 172.21.15.143:36446) 2024-03-25T19:35:24.535 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.534+0000 7fde0dcc1640 1 -- 172.21.15.143:0/2885215667 learned_addr learned my addr 172.21.15.143:0/2885215667 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:24.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.552+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379750965 0 0) 0x7fde080b0a70 con 0x7fde080b4d60 2024-03-25T19:35:24.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.552+0000 7fde0ccbf640 10 monclient(hunting): _negotiate 2024-03-25T19:35:24.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.552+0000 7fde0ccbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:24.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.552+0000 7fde0ccbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:24.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.552+0000 7fde0ccbf640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:35:24.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.552+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf4003520 con 0x7fde080b4d60 2024-03-25T19:35:24.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.556+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106745264 0 0) 0x7fde080af940 con 0x7fde0814be90 2024-03-25T19:35:24.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.556+0000 7fde0ccbf640 10 monclient(hunting): _negotiate 2024-03-25T19:35:24.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.556+0000 7fde0ccbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:24.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.556+0000 7fde0ccbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:24.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.556+0000 7fde0ccbf640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:35:24.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.556+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde080b0a70 con 0x7fde0814be90 2024-03-25T19:35:24.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.562+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31270149 0 0) 0x7fde080b0d50 con 0x7fde080b1730 2024-03-25T19:35:24.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.562+0000 7fde0ccbf640 10 monclient(hunting): _negotiate 2024-03-25T19:35:24.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.562+0000 7fde0ccbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:24.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.562+0000 7fde0ccbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:24.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.562+0000 7fde0ccbf640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:35:24.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.562+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde080af940 con 0x7fde080b1730 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 336741277 0 0) 0x7fddf4003520 con 0x7fde080b4d60 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 >> v1:172.21.15.143:6790/0 conn(0x7fde080b1730 legacy=0x7fde080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 >> v1:172.21.15.143:6789/0 conn(0x7fde0814be90 legacy=0x7fde0814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 10 monclient: _finish_hunting 0 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 1 monclient: found mon.noname-c 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080b1030 con 0x7fde080b4d60 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 10 monclient: _finish_auth 0 2024-03-25T19:35:24.566 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 10 monclient: _check_auth_tickets 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fde04001690 con 0x7fde080b4d60 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ccbf640 10 monclient: dump: 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ff25640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:35:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.565+0000 7fde0ff25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:24.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.569+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde040017f0 con 0x7fde080b4d60 2024-03-25T19:35:24.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.569+0000 7fde0ccbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:24.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.569+0000 7fde0ccbf640 1 -- 172.21.15.143:0/2885215667 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fde04001c00 con 0x7fde080b4d60 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.569+0000 7fde0ff25640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.569+0000 7fde0ff25640 4 set_mon_vals no callback set 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.569+0000 7fde0ccbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.569+0000 7fde0ccbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.569+0000 7fde0ccbf640 10 monclient: dump: 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:24.571 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.570+0000 7fde0ff25640 10 monclient: shutdown 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.570+0000 7fde0ff25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.570+0000 7fde0ff25640 1 -- 172.21.15.143:0/2885215667 >> v1:172.21.15.203:6789/0 conn(0x7fde080b4d60 legacy=0x7fde0813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.570+0000 7fde0ff25640 1 -- 172.21.15.143:0/2885215667 shutdown_connections 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.571+0000 7fde0ff25640 1 -- 172.21.15.143:0/2885215667 >> 172.21.15.143:0/2885215667 conn(0x7fde080617a0 msgr2=0x7fde08061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.571+0000 7fde0ff25640 1 -- 172.21.15.143:0/2885215667 shutdown_connections 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.571+0000 7fde0ff25640 1 -- 172.21.15.143:0/2885215667 wait complete. 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 10 monclient: build_initial_monmap 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 10 monclient: monmap: 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:35:24.573 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 1 librados: starting msgr at 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 1 librados: starting objecter 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 1 Processor -- start 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 1 -- start start 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 1 librados: setting wanted keys 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 1 librados: calling monclient init 2024-03-25T19:35:24.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.572+0000 7fde0ff25640 10 monclient: init 2024-03-25T19:35:24.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 10 monclient(hunting): picked mon.noname-a con 0x7fde080b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 10 monclient(hunting): picked mon.noname-c con 0x7fde080b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 10 monclient(hunting): picked mon.noname-b con 0x7fde0814be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde08140d00 con 0x7fde080b1730 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde08141cc0 con 0x7fde0814be90 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde081d8470 con 0x7fde080b4d60 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:24.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.573+0000 7fde0ff25640 10 monclient(hunting): authenticate will time out at 6340.918258s 2024-03-25T19:35:24.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.590+0000 7fde0dcc1640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fde080b1730 0x7fde081405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:32932/0 (socket says 172.21.15.143:32932) 2024-03-25T19:35:24.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.590+0000 7fde0dcc1640 1 -- 172.21.15.143:0/3837892268 learned_addr learned my addr 172.21.15.143:0/3837892268 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:24.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.592+0000 7fddfe7fc640 10 client.?.objecter ms_handle_connect 0x7fde080b1730 2024-03-25T19:35:24.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.592+0000 7fddfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:24.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.597+0000 7fddfe7fc640 10 client.?.objecter ms_handle_connect 0x7fde0814be90 2024-03-25T19:35:24.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.597+0000 7fddfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:24.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.603+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719738021 0 0) 0x7fde08140d00 con 0x7fde080b1730 2024-03-25T19:35:24.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.603+0000 7fddfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:35:24.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.603+0000 7fddfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:24.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.603+0000 7fddfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:24.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.603+0000 7fddfe7fc640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:35:24.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.604+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddec004eb0 con 0x7fde080b1730 2024-03-25T19:35:24.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.604+0000 7fddfe7fc640 10 client.?.objecter ms_handle_connect 0x7fde080b4d60 2024-03-25T19:35:24.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.604+0000 7fddfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.624+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2927707434 0 0) 0x7fddec004eb0 con 0x7fde080b1730 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.624+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde08140d00 con 0x7fde080b1730 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.624+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fde04001bc0 con 0x7fde080b1730 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.624+0000 7fddfe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.624+0000 7fddfe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.624+0000 7fddfe7fc640 10 monclient(hunting): dump: 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:24.625 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:24.626 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:24.626 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:24.626 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:24.626 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:24.626 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:24.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.624+0000 7fddfe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:24.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1396921831 0 0) 0x7fde08140d00 con 0x7fde080b1730 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 >> v1:172.21.15.143:6790/0 conn(0x7fde080b4d60 legacy=0x7fde081d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 >> v1:172.21.15.203:6789/0 conn(0x7fde0814be90 legacy=0x7fde08134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 1 monclient: found mon.a 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde081d8600 con 0x7fde080b1730 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fddfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fde0ff25640 5 monclient: authenticate success, global_id 4470 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fde0ff25640 10 monclient: _renew_subs 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fde0ff25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:24.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.679+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0813b3c0 con 0x7fde080b1730 2024-03-25T19:35:24.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.680+0000 7fde0ff25640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:24.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.680+0000 7fde0ff25640 10 monclient: _renew_subs 2024-03-25T19:35:24.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.680+0000 7fde0ff25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:24.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.680+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde081425a0 con 0x7fde080b1730 2024-03-25T19:35:24.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.680+0000 7fde0ff25640 1 librados: init done 2024-03-25T19:35:24.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.681+0000 7fde0ff25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:24.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.681+0000 7fde0ff25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:24.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.681+0000 7fde0ff25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:24.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.681+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde080b2270 con 0x7fde080b1730 2024-03-25T19:35:24.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.977+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde040011f0 con 0x7fde080b1730 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.977+0000 7fddfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.977+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fde04004c70 con 0x7fde080b1730 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.977+0000 7fddfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.977+0000 7fddfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.977+0000 7fddfe7fc640 10 monclient: dump: 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:24.979 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:24.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.977+0000 7fddfffff640 4 set_mon_vals no callback set 2024-03-25T19:35:24.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.978+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fde04004e90 con 0x7fde080b1730 2024-03-25T19:35:24.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.979+0000 7fde0d4c0640 10 monclient: get_auth_request con 0x7fddec06dc00 auth_method 0 2024-03-25T19:35:24.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.979+0000 7fddfe7fc640 10 client.4470.objecter ms_handle_connect 0x7fddec06dc00 2024-03-25T19:35:24.987 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.986+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fde040549c0 con 0x7fde080b1730 2024-03-25T19:35:24.987 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.986+0000 7fddfe7fc640 10 client.4470.objecter ms_dispatch 0x7fde08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:24.987 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.986+0000 7fddfe7fc640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:24.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.986+0000 7fddfe7fc640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:24.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.987+0000 7fddfe7fc640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-25T19:35:24.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.987+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde04050300 con 0x7fde080b1730 2024-03-25T19:35:24.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.987+0000 7fddfe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:24.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:24.987+0000 7fddfe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:25.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.244+0000 7fde0ff25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:25.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.244+0000 7fde0ff25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:25.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.244+0000 7fde0ff25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:25.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.244+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde080627b0 con 0x7fde080b1730 2024-03-25T19:35:25.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.273+0000 7fddfe7fc640 1 -- 172.21.15.143:0/3837892268 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fde04052400 con 0x7fde080b1730 2024-03-25T19:35:25.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.273+0000 7fddfe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:25.274 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.273+0000 7fddfe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:25.274 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:35:25.274 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:25.276 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 10 librados: watch_flush enter 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 10 librados: watch_flush exit 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 10 client.4470.objecter successfully canceled tick 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 >> v1:172.21.15.143:6800/1292282666 conn(0x7fddec06dc00 legacy=0x7fddec0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 10 monclient: shutdown 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 >> v1:172.21.15.143:6789/0 conn(0x7fde080b1730 legacy=0x7fde081405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 shutdown_connections 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.275+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 >> 172.21.15.143:0/3837892268 conn(0x7fde080617a0 msgr2=0x7fde080b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.276+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 shutdown_connections 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.276+0000 7fde0ff25640 1 -- 172.21.15.143:0/3837892268 wait complete. 2024-03-25T19:35:25.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:25.276+0000 7fde0ff25640 1 librados: shutdown 2024-03-25T19:35:25.289 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:32.096 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:32.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.209+0000 7fa90f84a640 10 monclient: get_monmap_and_config 2024-03-25T19:35:32.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.209+0000 7fa90f84a640 10 monclient: build_initial_monmap 2024-03-25T19:35:32.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.209+0000 7fa90f84a640 10 monclient: monmap: 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:35:32.210755+0000 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:35:32.210755+0000 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:35:32.210 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.210+0000 7fa90f84a640 1 Processor -- start 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.210+0000 7fa90f84a640 1 -- start start 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.210+0000 7fa90f84a640 10 monclient: init 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 10 monclient(hunting): picked mon.noname-c con 0x7fa9080a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 10 monclient(hunting): picked mon.noname-b con 0x7fa90813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 10 monclient(hunting): picked mon.noname-a con 0x7fa9081430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa908142520 con 0x7fa9081430c0 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9080b28a0 con 0x7fa9080a98f0 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9080b2a30 con 0x7fa90813f710 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:32.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.211+0000 7fa90f84a640 10 monclient(hunting): authenticate will time out at 6348.555603s 2024-03-25T19:35:32.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.217+0000 7fa90cde5640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fa90813f710 0x7fa9081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:48632/0 (socket says 172.21.15.143:48632) 2024-03-25T19:35:32.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.217+0000 7fa90cde5640 1 -- 172.21.15.143:0/3412807765 learned_addr learned my addr 172.21.15.143:0/3412807765 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:32.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.230+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433269346 0 0) 0x7fa908142520 con 0x7fa9081430c0 2024-03-25T19:35:32.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.230+0000 7fa8fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:32.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.230+0000 7fa8fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:32.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.230+0000 7fa8fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:32.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.230+0000 7fa8fffff640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:35:32.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.230+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f4003520 con 0x7fa9081430c0 2024-03-25T19:35:32.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.231+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395048629 0 0) 0x7fa9080b2a30 con 0x7fa90813f710 2024-03-25T19:35:32.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.231+0000 7fa8fffff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:32.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.231+0000 7fa8fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:32.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.231+0000 7fa8fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:32.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.231+0000 7fa8fffff640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:35:32.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.231+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa908142520 con 0x7fa90813f710 2024-03-25T19:35:32.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378023907 0 0) 0x7fa8f4003520 con 0x7fa9081430c0 2024-03-25T19:35:32.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 >> v1:172.21.15.143:6790/0 conn(0x7fa90813f710 legacy=0x7fa9081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 >> v1:172.21.15.203:6789/0 conn(0x7fa9080a98f0 legacy=0x7fa9080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 1 monclient: found mon.noname-a 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9080b44c0 con 0x7fa9081430c0 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 10 monclient: _finish_auth 0 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa900001a10 con 0x7fa9081430c0 2024-03-25T19:35:32.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa90f84a640 5 monclient: authenticate success, global_id 4473 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa90f84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa8fffff640 10 monclient: dump: 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:32.243 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:32.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.242+0000 7fa90f84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.247+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa900001bf0 con 0x7fa9081430c0 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.247+0000 7fa8fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.247+0000 7fa8fffff640 1 -- 172.21.15.143:0/3412807765 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa900002000 con 0x7fa9081430c0 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.247+0000 7fa90f84a640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.247+0000 7fa90f84a640 4 set_mon_vals no callback set 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.247+0000 7fa8fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.247+0000 7fa8fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.247+0000 7fa8fffff640 10 monclient: dump: 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.247 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:32.248 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:32.248 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:32.248 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:32.248 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:32.248 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.248+0000 7fa90f84a640 10 monclient: shutdown 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.248+0000 7fa90f84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.248+0000 7fa90f84a640 1 -- 172.21.15.143:0/3412807765 >> v1:172.21.15.143:6789/0 conn(0x7fa9081430c0 legacy=0x7fa90814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.248+0000 7fa90f84a640 1 -- 172.21.15.143:0/3412807765 shutdown_connections 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.248+0000 7fa90f84a640 1 -- 172.21.15.143:0/3412807765 >> 172.21.15.143:0/3412807765 conn(0x7fa9080b7ea0 msgr2=0x7fa9080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.248+0000 7fa90f84a640 1 -- 172.21.15.143:0/3412807765 shutdown_connections 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.248+0000 7fa90f84a640 1 -- 172.21.15.143:0/3412807765 wait complete. 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 10 monclient: build_initial_monmap 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 10 monclient: monmap: 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:32.250 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 1 librados: starting msgr at 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 1 librados: starting objecter 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 1 Processor -- start 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 1 -- start start 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 1 librados: setting wanted keys 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 1 librados: calling monclient init 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.249+0000 7fa90f84a640 10 monclient: init 2024-03-25T19:35:32.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 10 monclient(hunting): picked mon.noname-b con 0x7fa9080a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 10 monclient(hunting): picked mon.noname-c con 0x7fa90813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 10 monclient(hunting): picked mon.noname-a con 0x7fa9081430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa908136a50 con 0x7fa9081430c0 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9080b79c0 con 0x7fa9080a98f0 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9080b7ca0 con 0x7fa90813f710 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:32.252 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.250+0000 7fa90f84a640 10 monclient(hunting): authenticate will time out at 6348.594032s 2024-03-25T19:35:32.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.252+0000 7fa90d5e6640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa9080a98f0 0x7fa908137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:38810/0 (socket says 172.21.15.143:38810) 2024-03-25T19:35:32.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.252+0000 7fa90d5e6640 1 -- 172.21.15.143:0/1804133127 learned_addr learned my addr 172.21.15.143:0/1804133127 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:32.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.254+0000 7fa8fdffb640 10 client.?.objecter ms_handle_connect 0x7fa9080a98f0 2024-03-25T19:35:32.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.254+0000 7fa8fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:32.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.259+0000 7fa8fdffb640 10 client.?.objecter ms_handle_connect 0x7fa9081430c0 2024-03-25T19:35:32.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.259+0000 7fa8fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:32.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.265+0000 7fa8fdffb640 10 client.?.objecter ms_handle_connect 0x7fa90813f710 2024-03-25T19:35:32.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.265+0000 7fa8fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:32.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.265+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051869525 0 0) 0x7fa9080b79c0 con 0x7fa9080a98f0 2024-03-25T19:35:32.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.266+0000 7fa8fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:32.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.266+0000 7fa8fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:32.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.266+0000 7fa8fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:32.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.266+0000 7fa8fdffb640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:35:32.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.266+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ec004eb0 con 0x7fa9080a98f0 2024-03-25T19:35:32.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.268+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568289926 0 0) 0x7fa908136a50 con 0x7fa9081430c0 2024-03-25T19:35:32.268 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.268+0000 7fa8fdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:32.268 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.268+0000 7fa8fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:32.268 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.268+0000 7fa8fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:32.268 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.268+0000 7fa8fdffb640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:35:32.268 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.268+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9080b79c0 con 0x7fa9081430c0 2024-03-25T19:35:32.276 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1489223104 0 0) 0x7fa9080b79c0 con 0x7fa9081430c0 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa908136a50 con 0x7fa9081430c0 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa900001f00 con 0x7fa9081430c0 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 10 monclient(hunting): dump: 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:32.277 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:32.278 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:32.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:32.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805248416 0 0) 0x7fa8ec004eb0 con 0x7fa9080a98f0 2024-03-25T19:35:32.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.277+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9080b79c0 con 0x7fa9080a98f0 2024-03-25T19:35:32.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.278+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa9040019d0 con 0x7fa9080a98f0 2024-03-25T19:35:32.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.278+0000 7fa8fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:32.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.278+0000 7fa8fdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:32.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.278+0000 7fa8fdffb640 10 monclient(hunting): dump: 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:32.279 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:32.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2939347864 0 0) 0x7fa908136a50 con 0x7fa9081430c0 2024-03-25T19:35:32.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 >> v1:172.21.15.143:6790/0 conn(0x7fa90813f710 legacy=0x7fa908135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 >> v1:172.21.15.203:6789/0 conn(0x7fa9080a98f0 legacy=0x7fa908137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 1 monclient: found mon.a 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908136ca0 con 0x7fa9081430c0 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa8fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 5 monclient: authenticate success, global_id 4476 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 10 monclient: _renew_subs 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa908136f80 con 0x7fa9081430c0 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:32.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 10 monclient: _renew_subs 2024-03-25T19:35:32.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:32.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9081360e0 con 0x7fa9081430c0 2024-03-25T19:35:32.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.282+0000 7fa90f84a640 1 librados: init done 2024-03-25T19:35:32.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.283+0000 7fa90f84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:32.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.283+0000 7fa90f84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:32.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.283+0000 7fa90f84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:32.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.283+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9080627b0 con 0x7fa9081430c0 2024-03-25T19:35:32.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.607+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa900002080 con 0x7fa9081430c0 2024-03-25T19:35:32.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.607+0000 7fa8fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:32.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.607+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa900003dc0 con 0x7fa9081430c0 2024-03-25T19:35:32.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.607+0000 7fa8fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.607+0000 7fa8fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.607+0000 7fa8fdffb640 10 monclient: dump: 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:32.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.607+0000 7fa8ff7fe640 4 set_mon_vals no callback set 2024-03-25T19:35:32.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.608+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fa900019d90 con 0x7fa9081430c0 2024-03-25T19:35:32.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.609+0000 7fa90d5e6640 10 monclient: get_auth_request con 0x7fa8ec06de90 auth_method 0 2024-03-25T19:35:32.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.609+0000 7fa8fdffb640 10 client.4476.objecter ms_handle_connect 0x7fa8ec06de90 2024-03-25T19:35:32.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.615+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fa9000538f0 con 0x7fa9081430c0 2024-03-25T19:35:32.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.615+0000 7fa8fdffb640 10 client.4476.objecter ms_dispatch 0x7fa908000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:32.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.615+0000 7fa8fdffb640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:32.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.615+0000 7fa8fdffb640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:32.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.615+0000 7fa8fdffb640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-25T19:35:32.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.615+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa90004f230 con 0x7fa9081430c0 2024-03-25T19:35:32.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.615+0000 7fa8fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:32.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.615+0000 7fa8fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:32.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.868+0000 7fa90f84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:32.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.868+0000 7fa90f84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:32.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.868+0000 7fa90f84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:32.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.868+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa908135070 con 0x7fa9081430c0 2024-03-25T19:35:32.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.904+0000 7fa8fdffb640 1 -- 172.21.15.143:0/1804133127 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fa908135070 con 0x7fa9081430c0 2024-03-25T19:35:32.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.904+0000 7fa8fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:32.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.904+0000 7fa8fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:32.904 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:35:32.904 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:32.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 10 librados: watch_flush enter 2024-03-25T19:35:32.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 10 librados: watch_flush exit 2024-03-25T19:35:32.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-25T19:35:32.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 10 client.4476.objecter successfully canceled tick 2024-03-25T19:35:32.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 >> v1:172.21.15.143:6800/1292282666 conn(0x7fa8ec06de90 legacy=0x7fa8ec070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 10 monclient: shutdown 2024-03-25T19:35:32.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:32.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 >> v1:172.21.15.143:6789/0 conn(0x7fa9081430c0 legacy=0x7fa9081e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:32.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 shutdown_connections 2024-03-25T19:35:32.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 >> 172.21.15.143:0/1804133127 conn(0x7fa9080b7ea0 msgr2=0x7fa908074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:32.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 shutdown_connections 2024-03-25T19:35:32.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 1 -- 172.21.15.143:0/1804133127 wait complete. 2024-03-25T19:35:32.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:32.906+0000 7fa90f84a640 1 librados: shutdown 2024-03-25T19:35:32.919 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:38.763 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:05.327 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.770 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:05.335 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.813 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:05.378 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.815 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:05.351 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.820 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:05.385 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.820 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:05.356 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.823 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:05.388 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.842 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:05.378 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:38.862 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:05.427 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:39.725 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:39.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.827+0000 7f48774d6640 10 monclient: get_monmap_and_config 2024-03-25T19:35:39.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.827+0000 7f48774d6640 10 monclient: build_initial_monmap 2024-03-25T19:35:39.826 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.827+0000 7f48774d6640 10 monclient: monmap: 2024-03-25T19:35:39.826 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:35:39.827985+0000 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:35:39.827985+0000 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:35:39.827 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:39.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.827+0000 7f48774d6640 1 Processor -- start 2024-03-25T19:35:39.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 1 -- start start 2024-03-25T19:35:39.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 10 monclient: init 2024-03-25T19:35:39.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:39.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:35:39.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 10 monclient(hunting): picked mon.noname-b con 0x7f48700a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:39.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 10 monclient(hunting): picked mon.noname-a con 0x7f487013f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:39.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 10 monclient(hunting): picked mon.noname-c con 0x7f48701430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:39.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4870142520 con 0x7f487013f710 2024-03-25T19:35:39.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700b28a0 con 0x7f48701430c0 2024-03-25T19:35:39.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700b2a30 con 0x7f48700a98f0 2024-03-25T19:35:39.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:39.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.828+0000 7f48774d6640 10 monclient(hunting): authenticate will time out at 6356.172639s 2024-03-25T19:35:39.838 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.839+0000 7f4875272640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f48700a98f0 0x7f48700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:56642/0 (socket says 172.21.15.143:56642) 2024-03-25T19:35:39.838 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.839+0000 7f4875272640 1 -- 172.21.15.143:0/286443148 learned_addr learned my addr 172.21.15.143:0/286443148 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:39.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.849+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468931985 0 0) 0x7f48700b28a0 con 0x7f48701430c0 2024-03-25T19:35:39.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.849+0000 7f4867fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:39.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.849+0000 7f4867fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:39.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.849+0000 7f4867fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:39.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.849+0000 7f4867fff640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:35:39.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.849+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485c003520 con 0x7f48701430c0 2024-03-25T19:35:39.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.856+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110635850 0 0) 0x7f4870142520 con 0x7f487013f710 2024-03-25T19:35:39.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.856+0000 7f4867fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:39.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.856+0000 7f4867fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:39.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.856+0000 7f4867fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:39.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.856+0000 7f4867fff640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:35:39.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.856+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48700b28a0 con 0x7f487013f710 2024-03-25T19:35:39.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.857+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705849354 0 0) 0x7f48700b2a30 con 0x7f48700a98f0 2024-03-25T19:35:39.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.857+0000 7f4867fff640 10 monclient(hunting): _negotiate 2024-03-25T19:35:39.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.857+0000 7f4867fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:39.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.857+0000 7f4867fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:39.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.857+0000 7f4867fff640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:35:39.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.857+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4870142520 con 0x7f48700a98f0 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584807318 0 0) 0x7f485c003520 con 0x7f48701430c0 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 >> v1:172.21.15.143:6790/0 conn(0x7f48700a98f0 legacy=0x7f48700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 >> v1:172.21.15.143:6789/0 conn(0x7f487013f710 legacy=0x7f48701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 10 monclient: _finish_hunting 0 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 1 monclient: found mon.noname-c 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700b44c0 con 0x7f48701430c0 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 10 monclient: _finish_auth 0 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 10 monclient: _check_auth_tickets 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4868001a10 con 0x7f48701430c0 2024-03-25T19:35:39.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f48774d6640 5 monclient: authenticate success, global_id 4480 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f48774d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f4867fff640 10 monclient: dump: 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:39.868 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.867+0000 7f48774d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:39.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.878+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4868001b70 con 0x7f48701430c0 2024-03-25T19:35:39.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.878+0000 7f4867fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:39.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.878+0000 7f4867fff640 1 -- 172.21.15.143:0/286443148 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4868001f80 con 0x7f48701430c0 2024-03-25T19:35:39.877 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.878+0000 7f48774d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.878+0000 7f48774d6640 4 set_mon_vals no callback set 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.878+0000 7f4867fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.878+0000 7f4867fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.878+0000 7f4867fff640 10 monclient: dump: 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:39.878 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:39.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.879+0000 7f48774d6640 10 monclient: shutdown 2024-03-25T19:35:39.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.879+0000 7f48774d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.879+0000 7f48774d6640 1 -- 172.21.15.143:0/286443148 >> v1:172.21.15.203:6789/0 conn(0x7f48701430c0 legacy=0x7f487014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.879+0000 7f48774d6640 1 -- 172.21.15.143:0/286443148 shutdown_connections 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.879+0000 7f48774d6640 1 -- 172.21.15.143:0/286443148 >> 172.21.15.143:0/286443148 conn(0x7f48700b7ea0 msgr2=0x7f48700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.879+0000 7f48774d6640 1 -- 172.21.15.143:0/286443148 shutdown_connections 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.879+0000 7f48774d6640 1 -- 172.21.15.143:0/286443148 wait complete. 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.880+0000 7f48774d6640 10 monclient: build_initial_monmap 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.880+0000 7f48774d6640 10 monclient: monmap: 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:39.880 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:35:39.881 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:35:39.881 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.880+0000 7f48774d6640 1 librados: starting msgr at 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.880+0000 7f48774d6640 1 librados: starting objecter 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.880+0000 7f48774d6640 1 Processor -- start 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 1 -- start start 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 1 librados: setting wanted keys 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 1 librados: calling monclient init 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 10 monclient: init 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:39.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:35:39.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 10 monclient(hunting): picked mon.noname-a con 0x7f48700a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:39.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 10 monclient(hunting): picked mon.noname-b con 0x7f487013f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:39.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 10 monclient(hunting): picked mon.noname-c con 0x7f48701430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:39.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4870136ec0 con 0x7f48700a98f0 2024-03-25T19:35:39.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4870136c20 con 0x7f487013f710 2024-03-25T19:35:39.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4870137db0 con 0x7f48701430c0 2024-03-25T19:35:39.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:39.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.881+0000 7f48774d6640 10 monclient(hunting): authenticate will time out at 6356.225557s 2024-03-25T19:35:39.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.890+0000 7f4874a71640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f487013f710 0x7f48700b0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:44428/0 (socket says 172.21.15.143:44428) 2024-03-25T19:35:39.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.890+0000 7f4874a71640 1 -- 172.21.15.143:0/814957743 learned_addr learned my addr 172.21.15.143:0/814957743 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:39.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.892+0000 7f4865ffb640 10 client.?.objecter ms_handle_connect 0x7f487013f710 2024-03-25T19:35:39.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.892+0000 7f4865ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:39.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.901+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790127025 0 0) 0x7f4870136c20 con 0x7f487013f710 2024-03-25T19:35:39.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.901+0000 7f4865ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:39.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.901+0000 7f4865ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:39.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.901+0000 7f4865ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:39.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.901+0000 7f4865ffb640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:35:39.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.901+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4854004eb0 con 0x7f487013f710 2024-03-25T19:35:39.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.906+0000 7f4865ffb640 10 client.?.objecter ms_handle_connect 0x7f48700a98f0 2024-03-25T19:35:39.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.906+0000 7f4865ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:39.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.910+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652780570 0 0) 0x7f4870136ec0 con 0x7f48700a98f0 2024-03-25T19:35:39.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.910+0000 7f4865ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:39.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.910+0000 7f4865ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:39.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.910+0000 7f4865ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:39.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.910+0000 7f4865ffb640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:35:39.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.910+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4870136c20 con 0x7f48700a98f0 2024-03-25T19:35:39.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.913+0000 7f4865ffb640 10 client.?.objecter ms_handle_connect 0x7f48701430c0 2024-03-25T19:35:39.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.913+0000 7f4865ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:39.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.916+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2611272424 0 0) 0x7f4870136c20 con 0x7f48700a98f0 2024-03-25T19:35:39.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.916+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4870136ec0 con 0x7f48700a98f0 2024-03-25T19:35:39.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.917+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f486c0018b0 con 0x7f48700a98f0 2024-03-25T19:35:39.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.917+0000 7f4865ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:39.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.917+0000 7f4865ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.917+0000 7f4865ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:39.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.917+0000 7f4865ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:39.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.925+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815092848 0 0) 0x7f4870137db0 con 0x7f48701430c0 2024-03-25T19:35:39.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.925+0000 7f4865ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:39.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.925+0000 7f4865ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:39.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.925+0000 7f4865ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:39.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.925+0000 7f4865ffb640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:35:39.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.925+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4870136c20 con 0x7f48701430c0 2024-03-25T19:35:39.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.928+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1462171718 0 0) 0x7f4854004eb0 con 0x7f487013f710 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.929+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4870137db0 con 0x7f487013f710 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.929+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4860001a30 con 0x7f487013f710 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.929+0000 7f4865ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.929+0000 7f4865ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.929+0000 7f4865ffb640 10 monclient(hunting): dump: 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:39.929 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:39.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 71154438 0 0) 0x7f4870136ec0 con 0x7f48700a98f0 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 >> v1:172.21.15.143:6790/0 conn(0x7f48701430c0 legacy=0x7f48701e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 >> v1:172.21.15.203:6789/0 conn(0x7f487013f710 legacy=0x7f48700b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 1 monclient: found mon.a 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870138b50 con 0x7f48700a98f0 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f4865ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f48774d6640 5 monclient: authenticate success, global_id 4482 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f48774d6640 10 monclient: _renew_subs 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f48774d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:39.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4870139830 con 0x7f48700a98f0 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.935+0000 7f48774d6640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.936+0000 7f48774d6640 10 monclient: _renew_subs 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.936+0000 7f48774d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.936+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48701369b0 con 0x7f48700a98f0 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.936+0000 7f48774d6640 1 librados: init done 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.937+0000 7f48774d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.937+0000 7f48774d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.937+0000 7f48774d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:39.937 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.937+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487014f320 con 0x7f48700a98f0 2024-03-25T19:35:39.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.951+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f486c0047e0 con 0x7f48700a98f0 2024-03-25T19:35:39.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.951+0000 7f4865ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:39.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.951+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f486c001d80 con 0x7f48700a98f0 2024-03-25T19:35:39.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.951+0000 7f4865ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:39.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.951+0000 7f4865ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:39.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.951+0000 7f4865ffb640 10 monclient: dump: 2024-03-25T19:35:39.951 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:39.951 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:39.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.951+0000 7f48677fe640 4 set_mon_vals no callback set 2024-03-25T19:35:39.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.974+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f486c004bb0 con 0x7f48700a98f0 2024-03-25T19:35:39.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.975+0000 7f4874a71640 10 monclient: get_auth_request con 0x7f485406dfb0 auth_method 0 2024-03-25T19:35:39.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:39.975+0000 7f4865ffb640 10 client.4482.objecter ms_handle_connect 0x7f485406dfb0 2024-03-25T19:35:40.263 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.263+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f486c0547c0 con 0x7f48700a98f0 2024-03-25T19:35:40.263 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.263+0000 7f4865ffb640 10 client.4482.objecter ms_dispatch 0x7f4870000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:40.263 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.263+0000 7f4865ffb640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:40.263 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.263+0000 7f4865ffb640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:40.263 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.263+0000 7f4865ffb640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-25T19:35:40.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.263+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f486c050100 con 0x7f48700a98f0 2024-03-25T19:35:40.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.263+0000 7f4865ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:40.264 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.263+0000 7f4865ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:40.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.510+0000 7f48774d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:40.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.510+0000 7f48774d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:40.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.510+0000 7f48774d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:35:40.510 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.510+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48700627b0 con 0x7f48700a98f0 2024-03-25T19:35:40.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.539+0000 7f4865ffb640 1 -- 172.21.15.143:0/814957743 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f486c052200 con 0x7f48700a98f0 2024-03-25T19:35:40.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.539+0000 7f4865ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:40.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.539+0000 7f4865ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:40.539 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:35:40.539 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:40.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 10 librados: watch_flush enter 2024-03-25T19:35:40.541 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 10 librados: watch_flush exit 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 10 client.4482.objecter successfully canceled tick 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 >> v1:172.21.15.143:6800/1292282666 conn(0x7f485406dfb0 legacy=0x7f4854070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 10 monclient: shutdown 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 >> v1:172.21.15.143:6789/0 conn(0x7f48700a98f0 legacy=0x7f48701376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 shutdown_connections 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.541+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 >> 172.21.15.143:0/814957743 conn(0x7f48700b7ea0 msgr2=0x7f487013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.542+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 shutdown_connections 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.542+0000 7f48774d6640 1 -- 172.21.15.143:0/814957743 wait complete. 2024-03-25T19:35:40.542 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:40.542+0000 7f48774d6640 1 librados: shutdown 2024-03-25T19:35:40.553 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:47.360 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:47.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.476+0000 7f8eb8de6640 10 monclient: get_monmap_and_config 2024-03-25T19:35:47.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.476+0000 7f8eb8de6640 10 monclient: build_initial_monmap 2024-03-25T19:35:47.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.476+0000 7f8eb8de6640 10 monclient: monmap: 2024-03-25T19:35:47.476 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:47.476 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:47.476 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:35:47.477685+0000 2024-03-25T19:35:47.476 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:35:47.477685+0000 2024-03-25T19:35:47.476 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:47.477 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:47.477 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:47.477 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:35:47.477 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:35:47.477 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:47.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.477+0000 7f8eb8de6640 1 Processor -- start 2024-03-25T19:35:47.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.477+0000 7f8eb8de6640 1 -- start start 2024-03-25T19:35:47.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.477+0000 7f8eb8de6640 10 monclient: init 2024-03-25T19:35:47.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:47.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:47.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 10 monclient(hunting): picked mon.noname-a con 0x7f8eb40a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:47.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:47.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:47.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb4142520 con 0x7f8eb40a98f0 2024-03-25T19:35:47.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb40b28a0 con 0x7f8eb41430c0 2024-03-25T19:35:47.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb40b2a30 con 0x7f8eb413f710 2024-03-25T19:35:47.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:47.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.478+0000 7f8eb8de6640 10 monclient(hunting): authenticate will time out at 6363.822281s 2024-03-25T19:35:47.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.487+0000 7f8eb259c640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f8eb40a98f0 0x7f8eb40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:55756/0 (socket says 172.21.15.143:55756) 2024-03-25T19:35:47.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.487+0000 7f8eb259c640 1 -- 172.21.15.143:0/2223439851 learned_addr learned my addr 172.21.15.143:0/2223439851 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:47.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.497+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009791012 0 0) 0x7f8eb4142520 con 0x7f8eb40a98f0 2024-03-25T19:35:47.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.497+0000 7f8eb159a640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.497+0000 7f8eb159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.497+0000 7f8eb159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.497+0000 7f8eb159a640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:35:47.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.497+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9c003520 con 0x7f8eb40a98f0 2024-03-25T19:35:47.500 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.501+0000 7f8eb2d9d640 1 -- 172.21.15.143:0/2223439851 >> v1:172.21.15.203:6789/0 conn(0x7f8eb41430c0 legacy=0x7f8eb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T19:35:47.500 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.501+0000 7f8eb2d9d640 1 -- 172.21.15.143:0/2223439851 >> v1:172.21.15.203:6789/0 conn(0x7f8eb41430c0 legacy=0x7f8eb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:35:47.500 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.501+0000 7f8eb2d9d640 1 --1- 172.21.15.143:0/2223439851 >> v1:172.21.15.203:6789/0 conn(0x7f8eb41430c0 0x7f8eb414dc30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1733707062 0 0) 0x7f8e9c003520 con 0x7f8eb40a98f0 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 >> v1:172.21.15.143:6790/0 conn(0x7f8eb413f710 legacy=0x7f8eb41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 >> v1:172.21.15.203:6789/0 conn(0x7f8eb41430c0 legacy=0x7f8eb414dc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 10 monclient: _finish_hunting 0 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 1 monclient: found mon.noname-a 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb40b44c0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 10 monclient: _finish_auth 0 2024-03-25T19:35:47.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 10 monclient: _check_auth_tickets 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8ea4001690 con 0x7f8eb40a98f0 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb8de6640 5 monclient: authenticate success, global_id 4485 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb159a640 10 monclient: dump: 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:47.516 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:47.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.515+0000 7f8eb8de6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.534+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ea4001860 con 0x7f8eb40a98f0 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.534+0000 7f8eb159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.534+0000 7f8eb159a640 1 -- 172.21.15.143:0/2223439851 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8ea4001cb0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.534+0000 7f8eb8de6640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.534+0000 7f8eb8de6640 4 set_mon_vals no callback set 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.534+0000 7f8eb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.534+0000 7f8eb159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.534+0000 7f8eb159a640 10 monclient: dump: 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.534 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.535 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:47.535 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:47.535 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:47.535 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:47.535 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:47.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.535+0000 7f8eb8de6640 10 monclient: shutdown 2024-03-25T19:35:47.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.535+0000 7f8eb8de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:47.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.535+0000 7f8eb8de6640 1 -- 172.21.15.143:0/2223439851 >> v1:172.21.15.143:6789/0 conn(0x7f8eb40a98f0 legacy=0x7f8eb40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.535+0000 7f8eb8de6640 1 -- 172.21.15.143:0/2223439851 shutdown_connections 2024-03-25T19:35:47.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.535+0000 7f8eb8de6640 1 -- 172.21.15.143:0/2223439851 >> 172.21.15.143:0/2223439851 conn(0x7f8eb40b7ea0 msgr2=0x7f8eb40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:47.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.535+0000 7f8eb8de6640 1 -- 172.21.15.143:0/2223439851 shutdown_connections 2024-03-25T19:35:47.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.535+0000 7f8eb8de6640 1 -- 172.21.15.143:0/2223439851 wait complete. 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.536+0000 7f8eb8de6640 10 monclient: build_initial_monmap 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.536+0000 7f8eb8de6640 10 monclient: monmap: 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.536+0000 7f8eb8de6640 1 librados: starting msgr at 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.536+0000 7f8eb8de6640 1 librados: starting objecter 2024-03-25T19:35:47.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.537+0000 7f8eb8de6640 1 Processor -- start 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.537+0000 7f8eb8de6640 1 -- start start 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.537+0000 7f8eb8de6640 1 librados: setting wanted keys 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.537+0000 7f8eb8de6640 1 librados: calling monclient init 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.537+0000 7f8eb8de6640 10 monclient: init 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb40a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb413f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 10 monclient(hunting): picked mon.noname-a con 0x7f8eb41430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb40b6a20 con 0x7f8eb41430c0 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb41396f0 con 0x7f8eb413f710 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb413b260 con 0x7f8eb40a98f0 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:47.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.538+0000 7f8eb8de6640 10 monclient(hunting): authenticate will time out at 6363.881993s 2024-03-25T19:35:47.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.553+0000 7f8eb1d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f8eb413f710 0x7f8eb4139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:50582/0 (socket says 172.21.15.143:50582) 2024-03-25T19:35:47.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.553+0000 7f8eb1d9b640 1 -- 172.21.15.143:0/3525572838 learned_addr learned my addr 172.21.15.143:0/3525572838 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:47.554 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.554+0000 7f8ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f8eb40a98f0 2024-03-25T19:35:47.554 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.554+0000 7f8ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:47.556 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.556+0000 7f8ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f8eb413f710 2024-03-25T19:35:47.556 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.556+0000 7f8ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:47.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.558+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023797971 0 0) 0x7f8eb413b260 con 0x7f8eb40a98f0 2024-03-25T19:35:47.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.558+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.558+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.558+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.558+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:35:47.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.558+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e94004eb0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.565+0000 7f8ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f8eb41430c0 2024-03-25T19:35:47.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.565+0000 7f8ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:47.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.567+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930704437 0 0) 0x7f8eb41396f0 con 0x7f8eb413f710 2024-03-25T19:35:47.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.567+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.567+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.567+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.567+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:35:47.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.567+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb413b260 con 0x7f8eb413f710 2024-03-25T19:35:47.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.579+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942648568 0 0) 0x7f8eac003b20 con 0x7f8eb41430c0 2024-03-25T19:35:47.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.579+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:35:47.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.579+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:47.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.579+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:47.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.579+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:35:47.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.579+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb41396f0 con 0x7f8eb41430c0 2024-03-25T19:35:47.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.580+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1540961910 0 0) 0x7f8eb413b260 con 0x7f8eb413f710 2024-03-25T19:35:47.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.580+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eac003b20 con 0x7f8eb413f710 2024-03-25T19:35:47.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.580+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8ea80019b0 con 0x7f8eb413f710 2024-03-25T19:35:47.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.580+0000 7f8ea2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.580+0000 7f8ea2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:35:47.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.580+0000 7f8ea2ffd640 10 monclient(hunting): dump: 2024-03-25T19:35:47.580 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:47.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.580+0000 7f8ea2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.583+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 91635101 0 0) 0x7f8e94004eb0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.583+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eb413b260 con 0x7f8eb40a98f0 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.583+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8ea4001b70 con 0x7f8eb40a98f0 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.583+0000 7f8ea2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.583+0000 7f8ea2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.583+0000 7f8ea2ffd640 10 monclient(hunting): dump: 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:47.583 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:47.584 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:47.584 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:47.584 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 531401188 0 0) 0x7f8eb413b260 con 0x7f8eb40a98f0 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 >> v1:172.21.15.203:6789/0 conn(0x7f8eb413f710 legacy=0x7f8eb4139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 >> v1:172.21.15.143:6789/0 conn(0x7f8eb41430c0 legacy=0x7f8eb41e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 1 monclient: found mon.c 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb413b4d0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 5 monclient: authenticate success, global_id 4478 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 10 monclient: _renew_subs 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb413b7b0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 10 monclient: _renew_subs 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb40b70b0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.597+0000 7f8eb8de6640 1 librados: init done 2024-03-25T19:35:47.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.598+0000 7f8eb8de6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:47.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.598+0000 7f8eb8de6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:47.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.598+0000 7f8eb8de6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:47.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.598+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb40b3ef0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.916+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ea4001010 con 0x7f8eb40a98f0 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.916+0000 7f8ea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.916+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8ea4004bf0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.916+0000 7f8ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.916+0000 7f8ea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.916+0000 7f8ea2ffd640 10 monclient: dump: 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.916 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:47.917 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:47.917 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:47.917 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:47.917 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:47.917 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:47.917 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:47.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.916+0000 7f8eb0d99640 4 set_mon_vals no callback set 2024-03-25T19:35:47.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.917+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f8ea401ac00 con 0x7f8eb40a98f0 2024-03-25T19:35:47.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.918+0000 7f8eb1d9b640 10 monclient: get_auth_request con 0x7f8e9406df10 auth_method 0 2024-03-25T19:35:47.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.918+0000 7f8ea2ffd640 10 client.4478.objecter ms_handle_connect 0x7f8e9406df10 2024-03-25T19:35:47.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.923+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f8ea40548e0 con 0x7f8eb40a98f0 2024-03-25T19:35:47.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.923+0000 7f8ea2ffd640 10 client.4478.objecter ms_dispatch 0x7f8eb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:47.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.923+0000 7f8ea2ffd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:47.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.923+0000 7f8ea2ffd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:47.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.923+0000 7f8ea2ffd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T19:35:47.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.923+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ea4050220 con 0x7f8eb40a98f0 2024-03-25T19:35:47.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.923+0000 7f8ea2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:47.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:47.923+0000 7f8ea2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:48.138 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.138+0000 7f8eb8de6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:48.138 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.138+0000 7f8eb8de6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:48.138 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.138+0000 7f8eb8de6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:48.138 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.138+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8eb40627b0 con 0x7f8eb40a98f0 2024-03-25T19:35:48.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.243+0000 7f8ea2ffd640 1 -- 172.21.15.143:0/3525572838 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f8ea4052320 con 0x7f8eb40a98f0 2024-03-25T19:35:48.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.243+0000 7f8ea2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:48.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.243+0000 7f8ea2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:48.243 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:35:48.243 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:48.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 10 librados: watch_flush enter 2024-03-25T19:35:48.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 10 librados: watch_flush exit 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 10 client.4478.objecter successfully canceled tick 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 >> v1:172.21.15.143:6800/1292282666 conn(0x7f8e9406df10 legacy=0x7f8e940703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 10 monclient: shutdown 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 >> v1:172.21.15.143:6790/0 conn(0x7f8eb40a98f0 legacy=0x7f8eb412f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 shutdown_connections 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 >> 172.21.15.143:0/3525572838 conn(0x7f8eb40b7ea0 msgr2=0x7f8eb4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 shutdown_connections 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.245+0000 7f8eb8de6640 1 -- 172.21.15.143:0/3525572838 wait complete. 2024-03-25T19:35:48.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:48.246+0000 7f8eb8de6640 1 librados: shutdown 2024-03-25T19:35:48.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:55.064 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.178+0000 7fbdad7fd640 10 monclient: get_monmap_and_config 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.178+0000 7fbdad7fd640 10 monclient: build_initial_monmap 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.178+0000 7fbdad7fd640 10 monclient: monmap: 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:35:55.179125+0000 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:35:55.179125+0000 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:35:55.178 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:35:55.179 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:55.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.178+0000 7fbdad7fd640 1 Processor -- start 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 1 -- start start 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 10 monclient: init 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 10 monclient(hunting): picked mon.noname-a con 0x7fbda80a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 10 monclient(hunting): picked mon.noname-b con 0x7fbda813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 10 monclient(hunting): picked mon.noname-c con 0x7fbda81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda8142520 con 0x7fbda80a98f0 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda80b28a0 con 0x7fbda81430c0 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda80b2a30 con 0x7fbda813f710 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:55.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.179+0000 7fbdad7fd640 10 monclient(hunting): authenticate will time out at 6371.523445s 2024-03-25T19:35:55.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.192+0000 7fbda6ffd640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fbda80a98f0 0x7fbda80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:60666/0 (socket says 172.21.15.143:60666) 2024-03-25T19:35:55.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.192+0000 7fbda6ffd640 1 -- 172.21.15.143:0/140769416 learned_addr learned my addr 172.21.15.143:0/140769416 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:55.205 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.205+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545855682 0 0) 0x7fbda8142520 con 0x7fbda80a98f0 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.205+0000 7fbda5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.205+0000 7fbda5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.205+0000 7fbda5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.205+0000 7fbda5ffb640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.205+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd90003520 con 0x7fbda80a98f0 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.206+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232806639 0 0) 0x7fbda80b2a30 con 0x7fbda813f710 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.206+0000 7fbda5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.206+0000 7fbda5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.206+0000 7fbda5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.206+0000 7fbda5ffb640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:35:55.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.206+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda8142520 con 0x7fbda813f710 2024-03-25T19:35:55.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.211+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910979358 0 0) 0x7fbda80b28a0 con 0x7fbda81430c0 2024-03-25T19:35:55.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.211+0000 7fbda5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:35:55.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.211+0000 7fbda5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:55.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.211+0000 7fbda5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:55.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.211+0000 7fbda5ffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:35:55.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.211+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda80b2a30 con 0x7fbda81430c0 2024-03-25T19:35:55.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725790441 0 0) 0x7fbda80b2a30 con 0x7fbda81430c0 2024-03-25T19:35:55.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 >> v1:172.21.15.143:6790/0 conn(0x7fbda813f710 legacy=0x7fbda81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 >> v1:172.21.15.143:6789/0 conn(0x7fbda80a98f0 legacy=0x7fbda80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80b44c0 con 0x7fbda81430c0 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbda0001930 con 0x7fbda81430c0 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbda5ffb640 10 monclient: dump: 2024-03-25T19:35:55.225 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbdad7fd640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:35:55.226 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.224+0000 7fbdad7fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.243+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbda0001a90 con 0x7fbda81430c0 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.243+0000 7fbda5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.243+0000 7fbda5ffb640 1 -- 172.21.15.143:0/140769416 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbda0001ea0 con 0x7fbda81430c0 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.243+0000 7fbda5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.243+0000 7fbda5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.243+0000 7fbda5ffb640 10 monclient: dump: 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:55.243 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:55.244 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:55.244 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:55.244 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:55.244 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:55.244 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:55.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.243+0000 7fbdad7fd640 10 monclient: get_monmap_and_config success 2024-03-25T19:35:55.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.243+0000 7fbdad7fd640 4 set_mon_vals no callback set 2024-03-25T19:35:55.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.244+0000 7fbdad7fd640 10 monclient: shutdown 2024-03-25T19:35:55.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.244+0000 7fbdad7fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:55.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.244+0000 7fbdad7fd640 1 -- 172.21.15.143:0/140769416 >> v1:172.21.15.203:6789/0 conn(0x7fbda81430c0 legacy=0x7fbda814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:55.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.244+0000 7fbdad7fd640 1 -- 172.21.15.143:0/140769416 shutdown_connections 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.244+0000 7fbdad7fd640 1 -- 172.21.15.143:0/140769416 >> 172.21.15.143:0/140769416 conn(0x7fbda80b7ea0 msgr2=0x7fbda80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.245+0000 7fbdad7fd640 1 -- 172.21.15.143:0/140769416 shutdown_connections 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.245+0000 7fbdad7fd640 1 -- 172.21.15.143:0/140769416 wait complete. 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 10 monclient: build_initial_monmap 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 10 monclient: monmap: 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:35:55.246 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:35:55.247 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:35:55.247 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 1 librados: starting msgr at 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 1 librados: starting objecter 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 1 Processor -- start 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 1 -- start start 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 1 librados: setting wanted keys 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 1 librados: calling monclient init 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.246+0000 7fbdad7fd640 10 monclient: init 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 10 monclient: _reopen_session rank -1 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 10 monclient(hunting): picked mon.noname-b con 0x7fbda80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 10 monclient(hunting): picked mon.noname-a con 0x7fbda813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 10 monclient(hunting): picked mon.noname-c con 0x7fbda81430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:35:55.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda81395e0 con 0x7fbda813f710 2024-03-25T19:35:55.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda81398c0 con 0x7fbda80a98f0 2024-03-25T19:35:55.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbda80b0cc0 con 0x7fbda81430c0 2024-03-25T19:35:55.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 10 monclient(hunting): _renew_subs 2024-03-25T19:35:55.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.247+0000 7fbdad7fd640 10 monclient(hunting): authenticate will time out at 6371.591478s 2024-03-25T19:35:55.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.258+0000 7fbda77fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fbda81430c0 0x7fbda81d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:39984/0 (socket says 172.21.15.143:39984) 2024-03-25T19:35:55.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.258+0000 7fbda77fe640 1 -- 172.21.15.143:0/807096183 learned_addr learned my addr 172.21.15.143:0/807096183 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:35:55.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.261+0000 7fbd8f7fe640 10 client.?.objecter ms_handle_connect 0x7fbda81430c0 2024-03-25T19:35:55.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.261+0000 7fbd8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:55.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.266+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337179519 0 0) 0x7fbda80b0cc0 con 0x7fbda81430c0 2024-03-25T19:35:55.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.266+0000 7fbd8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:55.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.266+0000 7fbd8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:55.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.266+0000 7fbd8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:55.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.266+0000 7fbd8f7fe640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:35:55.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.266+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd94004eb0 con 0x7fbda81430c0 2024-03-25T19:35:55.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.266+0000 7fbd8f7fe640 10 client.?.objecter ms_handle_connect 0x7fbda80a98f0 2024-03-25T19:35:55.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.266+0000 7fbd8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:55.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.271+0000 7fbd8f7fe640 10 client.?.objecter ms_handle_connect 0x7fbda813f710 2024-03-25T19:35:55.270 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.271+0000 7fbd8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:35:55.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.273+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507200012 0 0) 0x7fbda81398c0 con 0x7fbda80a98f0 2024-03-25T19:35:55.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.273+0000 7fbd8f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:35:55.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.273+0000 7fbd8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:35:55.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.273+0000 7fbd8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:35:55.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.273+0000 7fbd8f7fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:35:55.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.273+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda80b0cc0 con 0x7fbda80a98f0 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.276+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3875584913 0 0) 0x7fbd94004eb0 con 0x7fbda81430c0 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.276+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbda81398c0 con 0x7fbda81430c0 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.276+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbda0001ea0 con 0x7fbda81430c0 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.276+0000 7fbd8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.276+0000 7fbd8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.276+0000 7fbd8f7fe640 10 monclient(hunting): dump: 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:55.276 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:55.277 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:55.277 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:55.277 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:55.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.276+0000 7fbd8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:35:55.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3616820495 0 0) 0x7fbda81398c0 con 0x7fbda81430c0 2024-03-25T19:35:55.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 >> v1:172.21.15.203:6789/0 conn(0x7fbda80a98f0 legacy=0x7fbda81d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:55.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 >> v1:172.21.15.143:6789/0 conn(0x7fbda813f710 legacy=0x7fbda80affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:55.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 1 monclient: found mon.c 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80b0e50 con 0x7fbda81430c0 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbd8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 5 monclient: authenticate success, global_id 4484 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 10 monclient: _renew_subs 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda80b06c0 con 0x7fbda81430c0 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:35:55.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 10 monclient: _renew_subs 2024-03-25T19:35:55.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:55.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda81d7b00 con 0x7fbda81430c0 2024-03-25T19:35:55.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.281+0000 7fbdad7fd640 1 librados: init done 2024-03-25T19:35:55.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.282+0000 7fbdad7fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:55.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.282+0000 7fbdad7fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:55.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.282+0000 7fbdad7fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:55.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.282+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda80b1060 con 0x7fbda81430c0 2024-03-25T19:35:55.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.574+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbda0003c20 con 0x7fbda81430c0 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.574+0000 7fbd8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.574+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbda0002100 con 0x7fbda81430c0 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.574+0000 7fbd8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.574+0000 7fbd8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.574+0000 7fbd8f7fe640 10 monclient: dump: 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:35:55.575 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:35:55.576 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:35:55.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.574+0000 7fbda57fa640 4 set_mon_vals no callback set 2024-03-25T19:35:55.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.575+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fbda0004070 con 0x7fbda81430c0 2024-03-25T19:35:55.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.575+0000 7fbda6ffd640 10 monclient: get_auth_request con 0x7fbd9406dd10 auth_method 0 2024-03-25T19:35:55.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.576+0000 7fbd8f7fe640 10 client.4484.objecter ms_handle_connect 0x7fbd9406dd10 2024-03-25T19:35:55.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.582+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fbda0054cd0 con 0x7fbda81430c0 2024-03-25T19:35:55.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.582+0000 7fbd8f7fe640 10 client.4484.objecter ms_dispatch 0x7fbda8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:35:55.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.582+0000 7fbd8f7fe640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:35:55.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.582+0000 7fbd8f7fe640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:35:55.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.582+0000 7fbd8f7fe640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-25T19:35:55.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.582+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbda0050610 con 0x7fbda81430c0 2024-03-25T19:35:55.582 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.582+0000 7fbd8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:35:55.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.582+0000 7fbd8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:35:55.823 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.823+0000 7fbdad7fd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:55.823 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.823+0000 7fbdad7fd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:55.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.823+0000 7fbdad7fd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:35:55.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.823+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbda80b3ef0 con 0x7fbda81430c0 2024-03-25T19:35:55.853 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.854+0000 7fbd8f7fe640 1 -- 172.21.15.143:0/807096183 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fbda0052710 con 0x7fbda81430c0 2024-03-25T19:35:55.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.854+0000 7fbd8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:35:55.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.854+0000 7fbd8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:35:55.854 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:35:55.854 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 10 librados: watch_flush enter 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 10 librados: watch_flush exit 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 10 client.4484.objecter successfully canceled tick 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 >> v1:172.21.15.143:6800/1292282666 conn(0x7fbd9406dd10 legacy=0x7fbd940701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 10 monclient: shutdown 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 >> v1:172.21.15.143:6790/0 conn(0x7fbda81430c0 legacy=0x7fbda81d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 shutdown_connections 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 >> 172.21.15.143:0/807096183 conn(0x7fbda80b7ea0 msgr2=0x7fbda8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 shutdown_connections 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.856+0000 7fbdad7fd640 1 -- 172.21.15.143:0/807096183 wait complete. 2024-03-25T19:35:55.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:35:55.857+0000 7fbdad7fd640 1 librados: shutdown 2024-03-25T19:35:55.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:35:58.837 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:25.402 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:58.842 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:25.407 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:58.887 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:25.452 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:58.909 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:25.445 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:58.916 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:25.480 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:58.916 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:25.452 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:58.924 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:25.489 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:58.939 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:25.475 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:35:58.949 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:25.514 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:02.674 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:02.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.787+0000 7faf3ffd7640 10 monclient: get_monmap_and_config 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.787+0000 7faf3ffd7640 10 monclient: build_initial_monmap 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.787+0000 7faf3ffd7640 10 monclient: monmap: 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:36:02.788207+0000 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:36:02.788207+0000 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:36:02.787 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.788+0000 7faf3ffd7640 1 Processor -- start 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.788+0000 7faf3ffd7640 1 -- start start 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.788+0000 7faf3ffd7640 10 monclient: init 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 10 monclient(hunting): picked mon.noname-c con 0x7faf380b10d0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 10 monclient(hunting): picked mon.noname-b con 0x7faf380b4360 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 10 monclient(hunting): picked mon.noname-a con 0x7faf380a9470 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf380a9ab0 con 0x7faf380a9470 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf380b2400 con 0x7faf380b10d0 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf380aa2a0 con 0x7faf380b4360 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:02.789 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.789+0000 7faf3ffd7640 10 monclient(hunting): authenticate will time out at 6379.132947s 2024-03-25T19:36:02.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.795+0000 7faf3dd73640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7faf380b10d0 0x7faf380b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:38234/0 (socket says 172.21.15.143:38234) 2024-03-25T19:36:02.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.795+0000 7faf3dd73640 1 -- 172.21.15.143:0/2043593482 learned_addr learned my addr 172.21.15.143:0/2043593482 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:02.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.817+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292206160 0 0) 0x7faf380b2400 con 0x7faf380b10d0 2024-03-25T19:36:02.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.817+0000 7faf3cd71640 10 monclient(hunting): _negotiate 2024-03-25T19:36:02.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.817+0000 7faf3cd71640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:02.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.817+0000 7faf3cd71640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:02.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.817+0000 7faf3cd71640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:36:02.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.817+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf24003520 con 0x7faf380b10d0 2024-03-25T19:36:02.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.820+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697332518 0 0) 0x7faf380a9ab0 con 0x7faf380a9470 2024-03-25T19:36:02.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.820+0000 7faf3cd71640 10 monclient(hunting): _negotiate 2024-03-25T19:36:02.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.820+0000 7faf3cd71640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:02.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.820+0000 7faf3cd71640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:02.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.820+0000 7faf3cd71640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:36:02.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.820+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf380b2400 con 0x7faf380a9470 2024-03-25T19:36:02.821 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.822+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610124373 0 0) 0x7faf380aa2a0 con 0x7faf380b4360 2024-03-25T19:36:02.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.822+0000 7faf3cd71640 10 monclient(hunting): _negotiate 2024-03-25T19:36:02.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.822+0000 7faf3cd71640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:02.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.822+0000 7faf3cd71640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:02.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.822+0000 7faf3cd71640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:36:02.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.822+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf380a9ab0 con 0x7faf380b4360 2024-03-25T19:36:02.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2912414144 0 0) 0x7faf24003520 con 0x7faf380b10d0 2024-03-25T19:36:02.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 >> v1:172.21.15.143:6790/0 conn(0x7faf380b4360 legacy=0x7faf380b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:02.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 >> v1:172.21.15.143:6789/0 conn(0x7faf380a9470 legacy=0x7faf3814dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:02.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 10 monclient: _finish_hunting 0 2024-03-25T19:36:02.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 1 monclient: found mon.noname-c 2024-03-25T19:36:02.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:02.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf380b14b0 con 0x7faf380b10d0 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 10 monclient: _finish_auth 0 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 10 monclient: _check_auth_tickets 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf34001690 con 0x7faf380b10d0 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3ffd7640 5 monclient: authenticate success, global_id 4495 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3ffd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3cd71640 10 monclient: dump: 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:02.835 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:02.836 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:02.836 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:02.836 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:02.836 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:02.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.834+0000 7faf3ffd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:02.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf340017f0 con 0x7faf380b10d0 2024-03-25T19:36:02.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3cd71640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:02.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3cd71640 1 -- 172.21.15.143:0/2043593482 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf34001c00 con 0x7faf380b10d0 2024-03-25T19:36:02.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3ffd7640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3ffd7640 4 set_mon_vals no callback set 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3cd71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3cd71640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3cd71640 10 monclient: dump: 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:02.847 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:02.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3ffd7640 10 monclient: shutdown 2024-03-25T19:36:02.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3ffd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:02.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.847+0000 7faf3ffd7640 1 -- 172.21.15.143:0/2043593482 >> v1:172.21.15.203:6789/0 conn(0x7faf380b10d0 legacy=0x7faf380b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:02.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.848+0000 7faf3ffd7640 1 -- 172.21.15.143:0/2043593482 shutdown_connections 2024-03-25T19:36:02.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.848+0000 7faf3ffd7640 1 -- 172.21.15.143:0/2043593482 >> 172.21.15.143:0/2043593482 conn(0x7faf380617a0 msgr2=0x7faf38061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.848+0000 7faf3ffd7640 1 -- 172.21.15.143:0/2043593482 shutdown_connections 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.848+0000 7faf3ffd7640 1 -- 172.21.15.143:0/2043593482 wait complete. 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 10 monclient: build_initial_monmap 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 10 monclient: monmap: 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 1 librados: starting msgr at 2024-03-25T19:36:02.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 1 librados: starting objecter 2024-03-25T19:36:02.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 1 Processor -- start 2024-03-25T19:36:02.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 1 -- start start 2024-03-25T19:36:02.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 1 librados: setting wanted keys 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 1 librados: calling monclient init 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.849+0000 7faf3ffd7640 10 monclient: init 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 10 monclient(hunting): picked mon.noname-a con 0x7faf380a9470 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 10 monclient(hunting): picked mon.noname-c con 0x7faf380b10d0 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 10 monclient(hunting): picked mon.noname-b con 0x7faf380b4360 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf38133320 con 0x7faf380a9470 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf381d7d40 con 0x7faf380b4360 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf381d7a20 con 0x7faf380b10d0 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:02.851 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.850+0000 7faf3ffd7640 10 monclient(hunting): authenticate will time out at 6379.193866s 2024-03-25T19:36:02.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.860+0000 7faf3dd73640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7faf380a9470 0x7faf381d7310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:60340/0 (socket says 172.21.15.143:60340) 2024-03-25T19:36:02.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.860+0000 7faf3dd73640 1 -- 172.21.15.143:0/4012890828 learned_addr learned my addr 172.21.15.143:0/4012890828 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:02.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.863+0000 7faf2e7fc640 10 client.?.objecter ms_handle_connect 0x7faf380a9470 2024-03-25T19:36:02.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.863+0000 7faf2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:02.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.864+0000 7faf2e7fc640 10 client.?.objecter ms_handle_connect 0x7faf380b4360 2024-03-25T19:36:02.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.864+0000 7faf2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:02.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.870+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521707462 0 0) 0x7faf381d7d40 con 0x7faf380b4360 2024-03-25T19:36:02.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.870+0000 7faf2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:02.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.870+0000 7faf2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:02.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.870+0000 7faf2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:02.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.870+0000 7faf2e7fc640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:36:02.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.870+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c004eb0 con 0x7faf380b4360 2024-03-25T19:36:02.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.875+0000 7faf2e7fc640 10 client.?.objecter ms_handle_connect 0x7faf380b10d0 2024-03-25T19:36:02.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.875+0000 7faf2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:02.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.878+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176923854 0 0) 0x7faf380b14b0 con 0x7faf380a9470 2024-03-25T19:36:02.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.878+0000 7faf2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:02.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.878+0000 7faf2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:02.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.878+0000 7faf2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:02.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.878+0000 7faf2e7fc640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:36:02.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.878+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf381d7d40 con 0x7faf380a9470 2024-03-25T19:36:02.886 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.887+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755351705 0 0) 0x7faf381d7a20 con 0x7faf380b10d0 2024-03-25T19:36:02.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.887+0000 7faf2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:02.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.887+0000 7faf2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:02.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.887+0000 7faf2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:02.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.887+0000 7faf2e7fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:36:02.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.887+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf380b14b0 con 0x7faf380b10d0 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.894+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3250484859 0 0) 0x7faf1c004eb0 con 0x7faf380b4360 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.894+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf381d7a20 con 0x7faf380b4360 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.894+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf30001bd0 con 0x7faf380b4360 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.894+0000 7faf2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.894+0000 7faf2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.894+0000 7faf2e7fc640 10 monclient(hunting): dump: 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:02.894 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.895 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.895 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:02.895 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:02.895 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:02.895 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:02.895 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:02.895 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:02.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.894+0000 7faf2e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.896+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4019010898 0 0) 0x7faf381d7d40 con 0x7faf380a9470 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.896+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf1c004eb0 con 0x7faf380a9470 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.896+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf340019a0 con 0x7faf380a9470 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.896+0000 7faf2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.896+0000 7faf2e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.896+0000 7faf2e7fc640 10 monclient(hunting): dump: 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.896 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:02.897 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:02.897 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:02.897 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:02.897 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:02.897 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:02.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.898+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 817735124 0 0) 0x7faf380b14b0 con 0x7faf380b10d0 2024-03-25T19:36:02.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.898+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf381d7d40 con 0x7faf380b10d0 2024-03-25T19:36:02.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.898+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf28001830 con 0x7faf380b10d0 2024-03-25T19:36:02.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.898+0000 7faf2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.898+0000 7faf2e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.898+0000 7faf2e7fc640 10 monclient(hunting): dump: 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:02.899 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:02.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4222712942 0 0) 0x7faf381d7a20 con 0x7faf380b4360 2024-03-25T19:36:02.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 >> v1:172.21.15.143:6790/0 conn(0x7faf380b10d0 legacy=0x7faf3813aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:02.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 >> v1:172.21.15.143:6789/0 conn(0x7faf380a9470 legacy=0x7faf381d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:02.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 1 monclient: found mon.b 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf381d81d0 con 0x7faf380b4360 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 10 monclient: _renew_subs 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf381d4ec0 con 0x7faf380b4360 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 10 monclient: _renew_subs 2024-03-25T19:36:02.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:02.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf381d45c0 con 0x7faf380b4360 2024-03-25T19:36:02.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.901+0000 7faf3ffd7640 1 librados: init done 2024-03-25T19:36:02.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.903+0000 7faf3ffd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:02.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.903+0000 7faf3ffd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:02.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.903+0000 7faf3ffd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:02.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:02.903+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf380627b0 con 0x7faf380b4360 2024-03-25T19:36:03.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.203+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf300020e0 con 0x7faf380b4360 2024-03-25T19:36:03.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.203+0000 7faf2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:03.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.203+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7faf30003b50 con 0x7faf380b4360 2024-03-25T19:36:03.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.203+0000 7faf2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:03.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.203+0000 7faf2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.203+0000 7faf2e7fc640 10 monclient: dump: 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:03.205 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.203+0000 7faf2f7fe640 4 set_mon_vals no callback set 2024-03-25T19:36:03.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.204+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7faf30019be0 con 0x7faf380b4360 2024-03-25T19:36:03.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.205+0000 7faf3dd73640 10 monclient: get_auth_request con 0x7faf1c06e0f0 auth_method 0 2024-03-25T19:36:03.206 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.205+0000 7faf2e7fc640 10 client.4498.objecter ms_handle_connect 0x7faf1c06e0f0 2024-03-25T19:36:03.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.210+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7faf30053870 con 0x7faf380b4360 2024-03-25T19:36:03.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.210+0000 7faf2e7fc640 10 client.4498.objecter ms_dispatch 0x7faf38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:03.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.210+0000 7faf2e7fc640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:03.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.210+0000 7faf2e7fc640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:03.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.210+0000 7faf2e7fc640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T19:36:03.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.211+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf3004f1b0 con 0x7faf380b4360 2024-03-25T19:36:03.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.211+0000 7faf2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:03.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.211+0000 7faf2e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:03.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.465+0000 7faf3ffd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:03.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.465+0000 7faf3ffd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:03.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.465+0000 7faf3ffd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:03.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.465+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf38145a90 con 0x7faf380b4360 2024-03-25T19:36:03.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.493+0000 7faf2e7fc640 1 -- 172.21.15.143:0/4012890828 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7faf300512b0 con 0x7faf380b4360 2024-03-25T19:36:03.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.493+0000 7faf2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:03.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.493+0000 7faf2e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:03.494 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:36:03.494 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.495+0000 7faf3ffd7640 10 librados: watch_flush enter 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.495+0000 7faf3ffd7640 10 librados: watch_flush exit 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.495+0000 7faf3ffd7640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.495+0000 7faf3ffd7640 10 client.4498.objecter successfully canceled tick 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.495+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 >> v1:172.21.15.143:6800/1292282666 conn(0x7faf1c06e0f0 legacy=0x7faf1c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.495+0000 7faf3ffd7640 10 monclient: shutdown 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.495+0000 7faf3ffd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.495+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 >> v1:172.21.15.203:6789/0 conn(0x7faf380b4360 legacy=0x7faf381e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.496+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 shutdown_connections 2024-03-25T19:36:03.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.496+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 >> 172.21.15.143:0/4012890828 conn(0x7faf380617a0 msgr2=0x7faf38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:03.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.496+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 shutdown_connections 2024-03-25T19:36:03.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.496+0000 7faf3ffd7640 1 -- 172.21.15.143:0/4012890828 wait complete. 2024-03-25T19:36:03.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:03.496+0000 7faf3ffd7640 1 librados: shutdown 2024-03-25T19:36:03.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:10.315 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:10.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.432+0000 7f473016d640 10 monclient: get_monmap_and_config 2024-03-25T19:36:10.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.432+0000 7f473016d640 10 monclient: build_initial_monmap 2024-03-25T19:36:10.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.432+0000 7f473016d640 10 monclient: monmap: 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:36:10.433818+0000 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:36:10.433818+0000 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:36:10.433 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.433+0000 7f473016d640 1 Processor -- start 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.434+0000 7f473016d640 1 -- start start 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.434+0000 7f473016d640 10 monclient: init 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.434+0000 7f473016d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.434+0000 7f473016d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.434+0000 7f473016d640 10 monclient(hunting): picked mon.noname-b con 0x7f47280a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.434+0000 7f473016d640 10 monclient(hunting): picked mon.noname-a con 0x7f472813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.435+0000 7f473016d640 10 monclient(hunting): picked mon.noname-c con 0x7f47281430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.435+0000 7f473016d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4728142520 con 0x7f472813f710 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.435+0000 7f473016d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47280b28a0 con 0x7f47281430c0 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.435+0000 7f473016d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47280b2a30 con 0x7f47280a98f0 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.435+0000 7f473016d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:10.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.435+0000 7f473016d640 10 monclient(hunting): authenticate will time out at 6386.778806s 2024-03-25T19:36:10.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.438+0000 7f472e70a640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f47281430c0 0x7f472814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:60324/0 (socket says 172.21.15.143:60324) 2024-03-25T19:36:10.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.439+0000 7f472e70a640 1 -- 172.21.15.143:0/1954829377 learned_addr learned my addr 172.21.15.143:0/1954829377 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:10.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.449+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400357419 0 0) 0x7f47280b28a0 con 0x7f47281430c0 2024-03-25T19:36:10.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.449+0000 7f472cf07640 10 monclient(hunting): _negotiate 2024-03-25T19:36:10.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.449+0000 7f472cf07640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:10.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.449+0000 7f472cf07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:10.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.449+0000 7f472cf07640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:36:10.450 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.449+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4714003520 con 0x7f47281430c0 2024-03-25T19:36:10.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.459+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887783001 0 0) 0x7f4728142520 con 0x7f472813f710 2024-03-25T19:36:10.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.460+0000 7f472cf07640 10 monclient(hunting): _negotiate 2024-03-25T19:36:10.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.460+0000 7f472cf07640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:10.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.460+0000 7f472cf07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:10.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.460+0000 7f472cf07640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:36:10.460 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.460+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47280b28a0 con 0x7f472813f710 2024-03-25T19:36:10.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133616491 0 0) 0x7f4714003520 con 0x7f47281430c0 2024-03-25T19:36:10.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 >> v1:172.21.15.143:6790/0 conn(0x7f47280a98f0 legacy=0x7f47280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:10.461 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 >> v1:172.21.15.143:6789/0 conn(0x7f472813f710 legacy=0x7f47281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 10 monclient: _finish_hunting 0 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 1 monclient: found mon.noname-c 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47280b44c0 con 0x7f47281430c0 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 10 monclient: _finish_auth 0 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 10 monclient: _check_auth_tickets 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4720001a10 con 0x7f47281430c0 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f472cf07640 10 monclient: dump: 2024-03-25T19:36:10.462 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f473016d640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:36:10.463 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.461+0000 7f473016d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:10.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4720001b70 con 0x7f47281430c0 2024-03-25T19:36:10.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f472cf07640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:10.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f472cf07640 1 -- 172.21.15.143:0/1954829377 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4720001f80 con 0x7f47281430c0 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f472cf07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f472cf07640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f472cf07640 10 monclient: dump: 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:10.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f473016d640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:10.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f473016d640 4 set_mon_vals no callback set 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f473016d640 10 monclient: shutdown 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f473016d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.470+0000 7f473016d640 1 -- 172.21.15.143:0/1954829377 >> v1:172.21.15.203:6789/0 conn(0x7f47281430c0 legacy=0x7f472814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.471+0000 7f473016d640 1 -- 172.21.15.143:0/1954829377 shutdown_connections 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.471+0000 7f473016d640 1 -- 172.21.15.143:0/1954829377 >> 172.21.15.143:0/1954829377 conn(0x7f47280b7ea0 msgr2=0x7f47280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.471+0000 7f473016d640 1 -- 172.21.15.143:0/1954829377 shutdown_connections 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.471+0000 7f473016d640 1 -- 172.21.15.143:0/1954829377 wait complete. 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 10 monclient: build_initial_monmap 2024-03-25T19:36:10.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 10 monclient: monmap: 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 1 librados: starting msgr at 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 1 librados: starting objecter 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 1 Processor -- start 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 1 -- start start 2024-03-25T19:36:10.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 1 librados: setting wanted keys 2024-03-25T19:36:10.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 1 librados: calling monclient init 2024-03-25T19:36:10.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.472+0000 7f473016d640 10 monclient: init 2024-03-25T19:36:10.474 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 10 monclient(hunting): picked mon.noname-b con 0x7f47280a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 10 monclient(hunting): picked mon.noname-c con 0x7f472813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 10 monclient(hunting): picked mon.noname-a con 0x7f47281430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472814e580 con 0x7f47281430c0 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472814e7d0 con 0x7f47280a98f0 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4728139990 con 0x7f472813f710 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:10.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.473+0000 7f473016d640 10 monclient(hunting): authenticate will time out at 6386.817348s 2024-03-25T19:36:10.487 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.487+0000 7f472df09640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f47280a98f0 0x7f472813c600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:60376/0 (socket says 172.21.15.143:60376) 2024-03-25T19:36:10.487 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.487+0000 7f472df09640 1 -- 172.21.15.143:0/1954899998 learned_addr learned my addr 172.21.15.143:0/1954899998 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:10.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.488+0000 7f471e7fc640 10 client.?.objecter ms_handle_connect 0x7f47280a98f0 2024-03-25T19:36:10.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.488+0000 7f471e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:10.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.496+0000 7f471e7fc640 10 client.?.objecter ms_handle_connect 0x7f47281430c0 2024-03-25T19:36:10.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.496+0000 7f471e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:10.497 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.498+0000 7f471e7fc640 10 client.?.objecter ms_handle_connect 0x7f472813f710 2024-03-25T19:36:10.498 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.498+0000 7f471e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:10.500 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.501+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860061149 0 0) 0x7f472814e7d0 con 0x7f47280a98f0 2024-03-25T19:36:10.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.501+0000 7f471e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:10.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.501+0000 7f471e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:10.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.501+0000 7f471e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:10.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.501+0000 7f471e7fc640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:36:10.501 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.501+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c004eb0 con 0x7f47280a98f0 2024-03-25T19:36:10.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.511+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801443398 0 0) 0x7f472814e580 con 0x7f47281430c0 2024-03-25T19:36:10.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.511+0000 7f471e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:10.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.511+0000 7f471e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:10.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.511+0000 7f471e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:10.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.511+0000 7f471e7fc640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:36:10.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.511+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f472814e7d0 con 0x7f47281430c0 2024-03-25T19:36:10.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.514+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453895856 0 0) 0x7f4728139990 con 0x7f472813f710 2024-03-25T19:36:10.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.514+0000 7f471e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:10.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.514+0000 7f471e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:10.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.514+0000 7f471e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:10.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.514+0000 7f471e7fc640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:36:10.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.514+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f472814e580 con 0x7f472813f710 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.539+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2650075211 0 0) 0x7f472814e7d0 con 0x7f47281430c0 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.539+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4728139990 con 0x7f47281430c0 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.539+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4720001e80 con 0x7f47281430c0 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.539+0000 7f471e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.539+0000 7f471e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.539+0000 7f471e7fc640 10 monclient(hunting): dump: 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:10.539 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:10.540 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:10.540 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:10.540 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:10.540 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:10.540 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:10.540 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:10.540 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:10.540 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.539+0000 7f471e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:10.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2931395423 0 0) 0x7f4728139990 con 0x7f47281430c0 2024-03-25T19:36:10.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 >> v1:172.21.15.143:6790/0 conn(0x7f472813f710 legacy=0x7f472813a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:10.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 >> v1:172.21.15.203:6789/0 conn(0x7f47280a98f0 legacy=0x7f472813c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:10.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 1 monclient: found mon.a 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472813bf00 con 0x7f47281430c0 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f471e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 5 monclient: authenticate success, global_id 4506 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 10 monclient: _renew_subs 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472813c1e0 con 0x7f47281430c0 2024-03-25T19:36:10.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:10.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 10 monclient: _renew_subs 2024-03-25T19:36:10.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:10.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f472813b070 con 0x7f47281430c0 2024-03-25T19:36:10.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.614+0000 7f473016d640 1 librados: init done 2024-03-25T19:36:10.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.615+0000 7f473016d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:10.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.615+0000 7f473016d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:10.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.615+0000 7f473016d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:10.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.615+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4728139690 con 0x7f47281430c0 2024-03-25T19:36:10.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.919+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4720001fe0 con 0x7f47281430c0 2024-03-25T19:36:10.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.919+0000 7f471e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:10.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.919+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4720004fb0 con 0x7f47281430c0 2024-03-25T19:36:10.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.919+0000 7f471e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:10.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.919+0000 7f471e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:10.919 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.919+0000 7f471e7fc640 10 monclient: dump: 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:10.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.919+0000 7f471ffff640 4 set_mon_vals no callback set 2024-03-25T19:36:10.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.920+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f472001af80 con 0x7f47281430c0 2024-03-25T19:36:10.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.921+0000 7f472df09640 10 monclient: get_auth_request con 0x7f470c06ddd0 auth_method 0 2024-03-25T19:36:10.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.921+0000 7f471e7fc640 10 client.4506.objecter ms_handle_connect 0x7f470c06ddd0 2024-03-25T19:36:10.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.927+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f4720054b00 con 0x7f47281430c0 2024-03-25T19:36:10.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.927+0000 7f471e7fc640 10 client.4506.objecter ms_dispatch 0x7f4728000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:10.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.927+0000 7f471e7fc640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:10.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.927+0000 7f471e7fc640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:10.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.928+0000 7f471e7fc640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-25T19:36:10.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.928+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4720050610 con 0x7f47281430c0 2024-03-25T19:36:10.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.928+0000 7f471e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:10.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:10.928+0000 7f471e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:11.181 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.181+0000 7f473016d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:11.181 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.181+0000 7f473016d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:11.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.181+0000 7f473016d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:11.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.181+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47280627b0 con 0x7f47281430c0 2024-03-25T19:36:11.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.205+0000 7f471e7fc640 1 -- 172.21.15.143:0/1954899998 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f4720052540 con 0x7f47281430c0 2024-03-25T19:36:11.204 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.205+0000 7f471e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:11.205 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.205+0000 7f471e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:11.205 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:36:11.205 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:11.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 10 librados: watch_flush enter 2024-03-25T19:36:11.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 10 librados: watch_flush exit 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 10 client.4506.objecter successfully canceled tick 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 >> v1:172.21.15.143:6800/1292282666 conn(0x7f470c06ddd0 legacy=0x7f470c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 10 monclient: shutdown 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 >> v1:172.21.15.143:6789/0 conn(0x7f47281430c0 legacy=0x7f47281e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 shutdown_connections 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 >> 172.21.15.143:0/1954899998 conn(0x7f47280b7ea0 msgr2=0x7f472814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 shutdown_connections 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 1 -- 172.21.15.143:0/1954899998 wait complete. 2024-03-25T19:36:11.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:11.207+0000 7f473016d640 1 librados: shutdown 2024-03-25T19:36:11.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:18.026 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:18.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.143+0000 7fa9744ff640 10 monclient: get_monmap_and_config 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.143+0000 7fa9744ff640 10 monclient: build_initial_monmap 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.143+0000 7fa9744ff640 10 monclient: monmap: 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:36:18.144188+0000 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:36:18.144188+0000 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:36:18.143 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 1 Processor -- start 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 1 -- start start 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 10 monclient: init 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 10 monclient(hunting): picked mon.noname-b con 0x7fa96c0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 10 monclient(hunting): picked mon.noname-a con 0x7fa96c13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 10 monclient(hunting): picked mon.noname-c con 0x7fa96c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.144+0000 7fa9744ff640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96c142520 con 0x7fa96c13f710 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.145+0000 7fa9744ff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96c0b28a0 con 0x7fa96c1430c0 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.145+0000 7fa9744ff640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96c0b2a30 con 0x7fa96c0a98f0 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.145+0000 7fa9744ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:18.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.145+0000 7fa9744ff640 10 monclient(hunting): authenticate will time out at 6394.488726s 2024-03-25T19:36:18.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.146+0000 7fa97229b640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fa96c0a98f0 0x7fa96c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:44912/0 (socket says 172.21.15.143:44912) 2024-03-25T19:36:18.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.146+0000 7fa97229b640 1 -- 172.21.15.143:0/2873545024 learned_addr learned my addr 172.21.15.143:0/2873545024 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:18.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.153+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705435975 0 0) 0x7fa96c0b2a30 con 0x7fa96c0a98f0 2024-03-25T19:36:18.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.154+0000 7fa971299640 10 monclient(hunting): _negotiate 2024-03-25T19:36:18.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.154+0000 7fa971299640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:18.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.154+0000 7fa971299640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:18.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.154+0000 7fa971299640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:36:18.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.154+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa958003520 con 0x7fa96c0a98f0 2024-03-25T19:36:18.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429784334 0 0) 0x7fa96c0b28a0 con 0x7fa96c1430c0 2024-03-25T19:36:18.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient(hunting): _negotiate 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa96c0b2a30 con 0x7fa96c1430c0 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3452815008 0 0) 0x7fa958003520 con 0x7fa96c0a98f0 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 >> v1:172.21.15.203:6789/0 conn(0x7fa96c1430c0 legacy=0x7fa96c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 >> v1:172.21.15.143:6789/0 conn(0x7fa96c13f710 legacy=0x7fa96c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient: _finish_hunting 0 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 1 monclient: found mon.noname-b 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c0b44c0 con 0x7fa96c0a98f0 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient: _finish_auth 0 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient: _check_auth_tickets 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:18.164 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa968001690 con 0x7fa96c0a98f0 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa971299640 10 monclient: dump: 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa9744ff640 5 monclient: authenticate success, global_id 4499 2024-03-25T19:36:18.165 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.163+0000 7fa9744ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:18.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.179+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa968001870 con 0x7fa96c0a98f0 2024-03-25T19:36:18.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.179+0000 7fa971299640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:18.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.179+0000 7fa971299640 1 -- 172.21.15.143:0/2873545024 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa968001c40 con 0x7fa96c0a98f0 2024-03-25T19:36:18.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.179+0000 7fa971299640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:18.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.179+0000 7fa971299640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:18.179 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.179+0000 7fa971299640 10 monclient: dump: 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.179+0000 7fa9744ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:18.180 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.179+0000 7fa9744ff640 4 set_mon_vals no callback set 2024-03-25T19:36:18.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.180+0000 7fa9744ff640 10 monclient: shutdown 2024-03-25T19:36:18.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.180+0000 7fa9744ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:18.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.180+0000 7fa9744ff640 1 -- 172.21.15.143:0/2873545024 >> v1:172.21.15.143:6790/0 conn(0x7fa96c0a98f0 legacy=0x7fa96c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:18.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.180+0000 7fa9744ff640 1 -- 172.21.15.143:0/2873545024 shutdown_connections 2024-03-25T19:36:18.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.180+0000 7fa9744ff640 1 -- 172.21.15.143:0/2873545024 >> 172.21.15.143:0/2873545024 conn(0x7fa96c0b7ea0 msgr2=0x7fa96c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:18.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.180+0000 7fa9744ff640 1 -- 172.21.15.143:0/2873545024 shutdown_connections 2024-03-25T19:36:18.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.180+0000 7fa9744ff640 1 -- 172.21.15.143:0/2873545024 wait complete. 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.181+0000 7fa9744ff640 10 monclient: build_initial_monmap 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.181+0000 7fa9744ff640 10 monclient: monmap: 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.181+0000 7fa9744ff640 1 librados: starting msgr at 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.181+0000 7fa9744ff640 1 librados: starting objecter 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.181+0000 7fa9744ff640 1 Processor -- start 2024-03-25T19:36:18.183 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.181+0000 7fa9744ff640 1 -- start start 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.181+0000 7fa9744ff640 1 librados: setting wanted keys 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.181+0000 7fa9744ff640 1 librados: calling monclient init 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 10 monclient: init 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 10 monclient(hunting): picked mon.noname-c con 0x7fa96c0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 10 monclient(hunting): picked mon.noname-a con 0x7fa96c13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 10 monclient(hunting): picked mon.noname-b con 0x7fa96c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96c138600 con 0x7fa96c13f710 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96c138b20 con 0x7fa96c1430c0 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96c0b1660 con 0x7fa96c0a98f0 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:18.184 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.182+0000 7fa9744ff640 10 monclient(hunting): authenticate will time out at 6394.526268s 2024-03-25T19:36:18.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.201+0000 7fa97229b640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fa96c0a98f0 0x7fa96c1377b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:44934/0 (socket says 172.21.15.143:44934) 2024-03-25T19:36:18.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.201+0000 7fa97229b640 1 -- 172.21.15.143:0/360396197 learned_addr learned my addr 172.21.15.143:0/360396197 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:18.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.203+0000 7fa962ffd640 10 client.?.objecter ms_handle_connect 0x7fa96c1430c0 2024-03-25T19:36:18.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.203+0000 7fa962ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:18.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.203+0000 7fa962ffd640 10 client.?.objecter ms_handle_connect 0x7fa96c0a98f0 2024-03-25T19:36:18.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.203+0000 7fa962ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:18.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.208+0000 7fa962ffd640 10 client.?.objecter ms_handle_connect 0x7fa96c13f710 2024-03-25T19:36:18.207 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.208+0000 7fa962ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:18.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.212+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178760540 0 0) 0x7fa96c138b20 con 0x7fa96c1430c0 2024-03-25T19:36:18.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.212+0000 7fa962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:18.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.212+0000 7fa962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:18.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.212+0000 7fa962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:18.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.212+0000 7fa962ffd640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:36:18.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.212+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa950004eb0 con 0x7fa96c1430c0 2024-03-25T19:36:18.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.221+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918935886 0 0) 0x7fa96c138600 con 0x7fa96c13f710 2024-03-25T19:36:18.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.221+0000 7fa962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:18.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.221+0000 7fa962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:18.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.221+0000 7fa962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:18.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.221+0000 7fa962ffd640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:36:18.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.221+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa96c138b20 con 0x7fa96c13f710 2024-03-25T19:36:18.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.224+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257751275 0 0) 0x7fa96c0b1660 con 0x7fa96c0a98f0 2024-03-25T19:36:18.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.224+0000 7fa962ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:18.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.224+0000 7fa962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:18.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.224+0000 7fa962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:18.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.224+0000 7fa962ffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:36:18.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.224+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa96c138600 con 0x7fa96c0a98f0 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.229+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1577070939 0 0) 0x7fa950004eb0 con 0x7fa96c1430c0 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.229+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa96c0b1660 con 0x7fa96c1430c0 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.229+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa964001bd0 con 0x7fa96c1430c0 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.229+0000 7fa962ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.229+0000 7fa962ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.229+0000 7fa962ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:18.229 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:18.230 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:18.230 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:18.230 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:18.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.229+0000 7fa962ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2416082750 0 0) 0x7fa96c0b1660 con 0x7fa96c1430c0 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 >> v1:172.21.15.143:6790/0 conn(0x7fa96c0a98f0 legacy=0x7fa96c1377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 >> v1:172.21.15.143:6789/0 conn(0x7fa96c13f710 legacy=0x7fa96c133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 1 monclient: found mon.b 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c134690 con 0x7fa96c1430c0 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.247+0000 7fa9744ff640 5 monclient: authenticate success, global_id 4510 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.248+0000 7fa9744ff640 10 monclient: _renew_subs 2024-03-25T19:36:18.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.248+0000 7fa9744ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.248+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96c1348e0 con 0x7fa96c1430c0 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.248+0000 7fa9744ff640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.248+0000 7fa9744ff640 10 monclient: _renew_subs 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.248+0000 7fa9744ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.248+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96c1381c0 con 0x7fa96c1430c0 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.248+0000 7fa9744ff640 1 librados: init done 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.249+0000 7fa9744ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.249+0000 7fa9744ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:18.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.249+0000 7fa9744ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:18.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.249+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96c138790 con 0x7fa96c1430c0 2024-03-25T19:36:18.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.530+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa964002060 con 0x7fa96c1430c0 2024-03-25T19:36:18.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.530+0000 7fa962ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:18.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.530+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fa964003b30 con 0x7fa96c1430c0 2024-03-25T19:36:18.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.530+0000 7fa962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:18.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.530+0000 7fa962ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:18.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.530+0000 7fa962ffd640 10 monclient: dump: 2024-03-25T19:36:18.530 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:18.531 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.530+0000 7fa970a98640 4 set_mon_vals no callback set 2024-03-25T19:36:18.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.531+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fa964019b40 con 0x7fa96c1430c0 2024-03-25T19:36:18.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.532+0000 7fa97229b640 10 monclient: get_auth_request con 0x7fa95006dd50 auth_method 0 2024-03-25T19:36:18.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.532+0000 7fa962ffd640 10 client.4510.objecter ms_handle_connect 0x7fa95006dd50 2024-03-25T19:36:18.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.536+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fa964053880 con 0x7fa96c1430c0 2024-03-25T19:36:18.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.536+0000 7fa962ffd640 10 client.4510.objecter ms_dispatch 0x7fa96c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:18.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.536+0000 7fa962ffd640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:18.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.536+0000 7fa962ffd640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:18.536 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.536+0000 7fa962ffd640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T19:36:18.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.537+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa96404f1c0 con 0x7fa96c1430c0 2024-03-25T19:36:18.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.537+0000 7fa962ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:18.537 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.537+0000 7fa962ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:18.768 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.768+0000 7fa9744ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:18.768 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.768+0000 7fa9744ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:18.768 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.768+0000 7fa9744ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:18.768 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.768+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa96c139a30 con 0x7fa96c1430c0 2024-03-25T19:36:18.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.791+0000 7fa962ffd640 1 -- 172.21.15.143:0/360396197 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fa9640512c0 con 0x7fa96c1430c0 2024-03-25T19:36:18.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.791+0000 7fa962ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:18.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.791+0000 7fa962ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:18.791 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:36:18.791 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 10 librados: watch_flush enter 2024-03-25T19:36:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 10 librados: watch_flush exit 2024-03-25T19:36:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T19:36:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 10 client.4510.objecter successfully canceled tick 2024-03-25T19:36:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 >> v1:172.21.15.143:6800/1292282666 conn(0x7fa95006dd50 legacy=0x7fa950070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 10 monclient: shutdown 2024-03-25T19:36:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:18.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 >> v1:172.21.15.203:6789/0 conn(0x7fa96c1430c0 legacy=0x7fa96c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 shutdown_connections 2024-03-25T19:36:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 >> 172.21.15.143:0/360396197 conn(0x7fa96c0b7ea0 msgr2=0x7fa96c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 shutdown_connections 2024-03-25T19:36:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 1 -- 172.21.15.143:0/360396197 wait complete. 2024-03-25T19:36:18.795 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:18.794+0000 7fa9744ff640 1 librados: shutdown 2024-03-25T19:36:18.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:18.927 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:45.491 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.935 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:45.500 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.981 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:06:45.546 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.996 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:45.532 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:18.999 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:45.564 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:19.005 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:45.541 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:19.008 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:45.572 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:19.030 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:06:45.595 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:19.030 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:06:45.566 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:25.611 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:25.729 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.730+0000 7efe41ad3640 10 monclient: get_monmap_and_config 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.730+0000 7efe41ad3640 10 monclient: build_initial_monmap 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.730+0000 7efe41ad3640 10 monclient: monmap: 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:36:25.730997+0000 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:36:25.730997+0000 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:36:25.730 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.730+0000 7efe41ad3640 1 Processor -- start 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.730+0000 7efe41ad3640 1 -- start start 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.730+0000 7efe41ad3640 10 monclient: init 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 10 monclient(hunting): picked mon.noname-a con 0x7efe3c06ab80 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 10 monclient(hunting): picked mon.noname-b con 0x7efe3c06af60 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 10 monclient(hunting): picked mon.noname-c con 0x7efe3c0b00b0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c0b0500 con 0x7efe3c06ab80 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c13ffa0 con 0x7efe3c0b00b0 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c0af520 con 0x7efe3c06af60 2024-03-25T19:36:25.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:25.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.731+0000 7efe41ad3640 10 monclient(hunting): authenticate will time out at 6402.075395s 2024-03-25T19:36:25.747 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.748+0000 7efe3b7fe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7efe3c06ab80 0x7efe3c0ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:49000/0 (socket says 172.21.15.143:49000) 2024-03-25T19:36:25.747 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.748+0000 7efe3b7fe640 1 -- 172.21.15.143:0/1496496363 learned_addr learned my addr 172.21.15.143:0/1496496363 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:25.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.765+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412976385 0 0) 0x7efe3c0af520 con 0x7efe3c06af60 2024-03-25T19:36:25.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.765+0000 7efe3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:36:25.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.765+0000 7efe3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:25.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.765+0000 7efe3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:25.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.765+0000 7efe3a7fc640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:36:25.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.765+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe24003520 con 0x7efe3c06af60 2024-03-25T19:36:25.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.769+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3167632233 0 0) 0x7efe24003520 con 0x7efe3c06af60 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 >> v1:172.21.15.203:6789/0 conn(0x7efe3c0b00b0 legacy=0x7efe3c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 >> v1:172.21.15.143:6789/0 conn(0x7efe3c06ab80 legacy=0x7efe3c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c072890 con 0x7efe3c06af60 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe41ad3640 5 monclient: authenticate success, global_id 4505 2024-03-25T19:36:25.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.770+0000 7efe41ad3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:25.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efe30001690 con 0x7efe3c06af60 2024-03-25T19:36:25.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:25.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:25.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 10 monclient: dump: 2024-03-25T19:36:25.778 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:25.778 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:25.778 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:25.778 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe30001920 con 0x7efe3c06af60 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe41ad3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe41ad3640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 1 -- 172.21.15.143:0/1496496363 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efe30001c30 con 0x7efe3c06af60 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe41ad3640 4 set_mon_vals no callback set 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:25.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.778+0000 7efe3a7fc640 10 monclient: dump: 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:25.780 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:25.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.779+0000 7efe41ad3640 10 monclient: shutdown 2024-03-25T19:36:25.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.779+0000 7efe41ad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:25.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.779+0000 7efe41ad3640 1 -- 172.21.15.143:0/1496496363 >> v1:172.21.15.143:6790/0 conn(0x7efe3c06af60 legacy=0x7efe3c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:25.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.779+0000 7efe41ad3640 1 -- 172.21.15.143:0/1496496363 shutdown_connections 2024-03-25T19:36:25.782 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.779+0000 7efe41ad3640 1 -- 172.21.15.143:0/1496496363 >> 172.21.15.143:0/1496496363 conn(0x7efe3c0b4a80 msgr2=0x7efe3c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.779+0000 7efe41ad3640 1 -- 172.21.15.143:0/1496496363 shutdown_connections 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.779+0000 7efe41ad3640 1 -- 172.21.15.143:0/1496496363 wait complete. 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 10 monclient: build_initial_monmap 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 10 monclient: monmap: 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:36:25.783 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 1 librados: starting msgr at 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 1 librados: starting objecter 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 1 Processor -- start 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 1 -- start start 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 1 librados: setting wanted keys 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 1 librados: calling monclient init 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.780+0000 7efe41ad3640 10 monclient: init 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 10 monclient(hunting): picked mon.noname-b con 0x7efe3c06ab80 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 10 monclient(hunting): picked mon.noname-a con 0x7efe3c0b00b0 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 10 monclient(hunting): picked mon.noname-c con 0x7efe3c1442e0 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c145a00 con 0x7efe3c0b00b0 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c145300 con 0x7efe3c06ab80 2024-03-25T19:36:25.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c1d73b0 con 0x7efe3c1442e0 2024-03-25T19:36:25.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:25.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.781+0000 7efe41ad3640 10 monclient(hunting): authenticate will time out at 6402.125506s 2024-03-25T19:36:25.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.783+0000 7efe3affd640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7efe3c0b00b0 0x7efe3c145ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:49060/0 (socket says 172.21.15.143:49060) 2024-03-25T19:36:25.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.783+0000 7efe3affd640 1 -- 172.21.15.143:0/4235390043 learned_addr learned my addr 172.21.15.143:0/4235390043 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:25.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.786+0000 7efe23fff640 10 client.?.objecter ms_handle_connect 0x7efe3c0b00b0 2024-03-25T19:36:25.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.786+0000 7efe23fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:25.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.791+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911782968 0 0) 0x7efe3c145a00 con 0x7efe3c0b00b0 2024-03-25T19:36:25.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.791+0000 7efe23fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:25.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.791+0000 7efe23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:25.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.791+0000 7efe23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:25.791 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.791+0000 7efe23fff640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:36:25.792 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.791+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe28004eb0 con 0x7efe3c0b00b0 2024-03-25T19:36:25.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.802+0000 7efe23fff640 10 client.?.objecter ms_handle_connect 0x7efe3c06ab80 2024-03-25T19:36:25.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.802+0000 7efe23fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:25.803 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.803+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3960529806 0 0) 0x7efe28004eb0 con 0x7efe3c0b00b0 2024-03-25T19:36:25.803 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.803+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe3c145a00 con 0x7efe3c0b00b0 2024-03-25T19:36:25.803 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.803+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efe30001710 con 0x7efe3c0b00b0 2024-03-25T19:36:25.803 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.803+0000 7efe23fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:25.803 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.803+0000 7efe23fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:25.803 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.803+0000 7efe23fff640 10 monclient(hunting): dump: 2024-03-25T19:36:25.803 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:25.803 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:25.804 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.803+0000 7efe23fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:25.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.813+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212721860 0 0) 0x7efe3c145300 con 0x7efe3c06ab80 2024-03-25T19:36:25.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.813+0000 7efe23fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:25.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.813+0000 7efe23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:25.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.813+0000 7efe23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:25.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.813+0000 7efe23fff640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:36:25.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.813+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe28004eb0 con 0x7efe3c06ab80 2024-03-25T19:36:25.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.818+0000 7efe23fff640 10 client.?.objecter ms_handle_connect 0x7efe3c1442e0 2024-03-25T19:36:25.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.818+0000 7efe23fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:25.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.829+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283745480 0 0) 0x7efe3c1d73b0 con 0x7efe3c1442e0 2024-03-25T19:36:25.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.829+0000 7efe23fff640 10 monclient(hunting): _negotiate 2024-03-25T19:36:25.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.829+0000 7efe23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:25.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.829+0000 7efe23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:25.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.829+0000 7efe23fff640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:36:25.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.829+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c145300 con 0x7efe3c1442e0 2024-03-25T19:36:25.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.830+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4275196515 0 0) 0x7efe28004eb0 con 0x7efe3c06ab80 2024-03-25T19:36:25.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.830+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe3c1d73b0 con 0x7efe3c06ab80 2024-03-25T19:36:25.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.830+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efe2c001860 con 0x7efe3c06ab80 2024-03-25T19:36:25.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.830+0000 7efe23fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:25.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.830+0000 7efe23fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:25.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.830+0000 7efe23fff640 10 monclient(hunting): dump: 2024-03-25T19:36:25.830 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:25.831 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:25.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2385349516 0 0) 0x7efe3c145a00 con 0x7efe3c0b00b0 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 >> v1:172.21.15.143:6790/0 conn(0x7efe3c1442e0 legacy=0x7efe3c1d7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 >> v1:172.21.15.203:6789/0 conn(0x7efe3c06ab80 legacy=0x7efe3c13e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 10 monclient: _finish_hunting 0 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 1 monclient: found mon.a 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c1d7540 con 0x7efe3c0b00b0 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 10 monclient: _finish_auth 0 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 10 monclient: _check_auth_tickets 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 5 monclient: authenticate success, global_id 4515 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 10 monclient: _renew_subs 2024-03-25T19:36:25.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:25.837 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3c1465f0 con 0x7efe3c0b00b0 2024-03-25T19:36:25.837 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:25.837 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 10 monclient: _renew_subs 2024-03-25T19:36:25.837 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:25.837 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3c1d8250 con 0x7efe3c0b00b0 2024-03-25T19:36:25.837 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.835+0000 7efe41ad3640 1 librados: init done 2024-03-25T19:36:25.838 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.836+0000 7efe41ad3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:25.838 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.836+0000 7efe41ad3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:25.838 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.836+0000 7efe41ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:25.838 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:25.837+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3c0adf10 con 0x7efe3c0b00b0 2024-03-25T19:36:26.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.142+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe30003960 con 0x7efe3c0b00b0 2024-03-25T19:36:26.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.142+0000 7efe23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:26.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.142+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efe30003d30 con 0x7efe3c0b00b0 2024-03-25T19:36:26.142 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.142+0000 7efe23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.142+0000 7efe23fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.142+0000 7efe23fff640 10 monclient: dump: 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.142+0000 7efe39ffb640 4 set_mon_vals no callback set 2024-03-25T19:36:26.143 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.142+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7efe30003f50 con 0x7efe3c0b00b0 2024-03-25T19:36:26.144 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.143+0000 7efe3b7fe640 10 monclient: get_auth_request con 0x7efe2806df80 auth_method 0 2024-03-25T19:36:26.144 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.143+0000 7efe23fff640 10 client.4515.objecter ms_handle_connect 0x7efe2806df80 2024-03-25T19:36:26.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.149+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7efe30054a50 con 0x7efe3c0b00b0 2024-03-25T19:36:26.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.149+0000 7efe23fff640 10 client.4515.objecter ms_dispatch 0x7efe3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:26.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.149+0000 7efe23fff640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:26.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.149+0000 7efe23fff640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:26.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.149+0000 7efe23fff640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-25T19:36:26.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.149+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe30050560 con 0x7efe3c0b00b0 2024-03-25T19:36:26.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.149+0000 7efe23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:26.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.149+0000 7efe23fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:26.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.401+0000 7efe41ad3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:26.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.401+0000 7efe41ad3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:26.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.401+0000 7efe41ad3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:36:26.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.401+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe3c0627b0 con 0x7efe3c0b00b0 2024-03-25T19:36:26.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.438+0000 7efe23fff640 1 -- 172.21.15.143:0/4235390043 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7efe30052490 con 0x7efe3c0b00b0 2024-03-25T19:36:26.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.438+0000 7efe23fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:26.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.438+0000 7efe23fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:26.438 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:36:26.438 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:26.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 10 librados: watch_flush enter 2024-03-25T19:36:26.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 10 librados: watch_flush exit 2024-03-25T19:36:26.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-25T19:36:26.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 10 client.4515.objecter successfully canceled tick 2024-03-25T19:36:26.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 >> v1:172.21.15.143:6800/1292282666 conn(0x7efe2806df80 legacy=0x7efe28070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:26.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 10 monclient: shutdown 2024-03-25T19:36:26.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:26.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 >> v1:172.21.15.143:6789/0 conn(0x7efe3c0b00b0 legacy=0x7efe3c145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:26.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 shutdown_connections 2024-03-25T19:36:26.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 >> 172.21.15.143:0/4235390043 conn(0x7efe3c0b4a80 msgr2=0x7efe3c06f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:26.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 shutdown_connections 2024-03-25T19:36:26.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 1 -- 172.21.15.143:0/4235390043 wait complete. 2024-03-25T19:36:26.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:26.441+0000 7efe41ad3640 1 librados: shutdown 2024-03-25T19:36:26.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:33.260 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.379+0000 7f6cb4e40640 10 monclient: get_monmap_and_config 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.379+0000 7f6cb4e40640 10 monclient: build_initial_monmap 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.379+0000 7f6cb4e40640 10 monclient: monmap: 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:36:33.380462+0000 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:36:33.380462+0000 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:33.379 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:33.380 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:36:33.380 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:36:33.380 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.380+0000 7f6cb4e40640 1 Processor -- start 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.380+0000 7f6cb4e40640 1 -- start start 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.380+0000 7f6cb4e40640 10 monclient: init 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.380+0000 7f6cb4e40640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.380+0000 7f6cb4e40640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.381+0000 7f6cb4e40640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb00b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.381+0000 7f6cb4e40640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb00b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.381+0000 7f6cb4e40640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb014be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:33.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.381+0000 7f6cb4e40640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00af940 con 0x7f6cb00b1730 2024-03-25T19:36:33.382 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.381+0000 7f6cb4e40640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00b0a70 con 0x7f6cb00b4d60 2024-03-25T19:36:33.382 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.381+0000 7f6cb4e40640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb00b0d50 con 0x7f6cb014be90 2024-03-25T19:36:33.382 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.381+0000 7f6cb4e40640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:33.382 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.381+0000 7f6cb4e40640 10 monclient(hunting): authenticate will time out at 6409.724720s 2024-03-25T19:36:33.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.385+0000 7f6cae59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6cb00b4d60 0x7f6cb013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:50484/0 (socket says 172.21.15.143:50484) 2024-03-25T19:36:33.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.385+0000 7f6cae59c640 1 -- 172.21.15.143:0/3601309869 learned_addr learned my addr 172.21.15.143:0/3601309869 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:33.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.396+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968585666 0 0) 0x7f6cb00b0a70 con 0x7f6cb00b4d60 2024-03-25T19:36:33.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.396+0000 7f6cadd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:33.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.396+0000 7f6cadd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:33.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.396+0000 7f6cadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:33.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.396+0000 7f6cadd9b640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:36:33.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.396+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c003520 con 0x7f6cb00b4d60 2024-03-25T19:36:33.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.406+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030704999 0 0) 0x7f6cb00b0d50 con 0x7f6cb014be90 2024-03-25T19:36:33.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.406+0000 7f6cadd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:36:33.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.406+0000 7f6cadd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:33.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.406+0000 7f6cadd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:33.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.406+0000 7f6cadd9b640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:36:33.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.406+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb00b0a70 con 0x7f6cb014be90 2024-03-25T19:36:33.417 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2471140654 0 0) 0x7f6c9c003520 con 0x7f6cb00b4d60 2024-03-25T19:36:33.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 >> v1:172.21.15.143:6790/0 conn(0x7f6cb014be90 legacy=0x7f6cb014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:33.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 >> v1:172.21.15.143:6789/0 conn(0x7f6cb00b1730 legacy=0x7f6cb00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:33.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:36:33.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 1 monclient: found mon.noname-c 2024-03-25T19:36:33.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:33.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:36:33.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb00b1030 con 0x7f6cb00b4d60 2024-03-25T19:36:33.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 10 monclient: _finish_auth 0 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6ca0001690 con 0x7f6cb00b4d60 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.417+0000 7f6cadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.418+0000 7f6cadd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.418+0000 7f6cadd9b640 10 monclient: dump: 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:33.419 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:33.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.418+0000 7f6cb4e40640 5 monclient: authenticate success, global_id 4519 2024-03-25T19:36:33.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.418+0000 7f6cb4e40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:33.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.430+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca00017f0 con 0x7f6cb00b4d60 2024-03-25T19:36:33.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.430+0000 7f6cadd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:33.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.430+0000 7f6cadd9b640 1 -- 172.21.15.143:0/3601309869 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6ca0001c00 con 0x7f6cb00b4d60 2024-03-25T19:36:33.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.430+0000 7f6cb4e40640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:33.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.430+0000 7f6cb4e40640 4 set_mon_vals no callback set 2024-03-25T19:36:33.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.430+0000 7f6cadd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:33.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.430+0000 7f6cadd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:33.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.430+0000 7f6cadd9b640 10 monclient: dump: 2024-03-25T19:36:33.431 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:33.431 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:33.431 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.431 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.431 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:33.431 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:33.432 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:33.432 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:33.432 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:33.432 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:33.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.431+0000 7f6cb4e40640 10 monclient: shutdown 2024-03-25T19:36:33.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.431+0000 7f6cb4e40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:33.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.431+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3601309869 >> v1:172.21.15.203:6789/0 conn(0x7f6cb00b4d60 legacy=0x7f6cb013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:33.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.431+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3601309869 shutdown_connections 2024-03-25T19:36:33.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.431+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3601309869 >> 172.21.15.143:0/3601309869 conn(0x7f6cb00617a0 msgr2=0x7f6cb0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:33.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.431+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3601309869 shutdown_connections 2024-03-25T19:36:33.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.431+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3601309869 wait complete. 2024-03-25T19:36:33.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 10 monclient: build_initial_monmap 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 10 monclient: monmap: 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 1 librados: starting msgr at 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 1 librados: starting objecter 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 1 Processor -- start 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 1 -- start start 2024-03-25T19:36:33.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 1 librados: setting wanted keys 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 1 librados: calling monclient init 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.432+0000 7f6cb4e40640 10 monclient: init 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 10 monclient(hunting): picked mon.noname-c con 0x7f6cb00b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 10 monclient(hunting): picked mon.noname-a con 0x7f6cb00b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 10 monclient(hunting): picked mon.noname-b con 0x7f6cb014be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb0144150 con 0x7f6cb00b4d60 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb0145840 con 0x7f6cb014be90 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb0143c30 con 0x7f6cb00b1730 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:33.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.433+0000 7f6cb4e40640 10 monclient(hunting): authenticate will time out at 6409.777107s 2024-03-25T19:36:33.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.440+0000 7f6caed9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6cb014be90 0x7f6cb01e2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:50520/0 (socket says 172.21.15.143:50520) 2024-03-25T19:36:33.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.441+0000 7f6caed9d640 1 -- 172.21.15.143:0/3954037275 learned_addr learned my addr 172.21.15.143:0/3954037275 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:33.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.443+0000 7f6ca6ffd640 10 client.?.objecter ms_handle_connect 0x7f6cb014be90 2024-03-25T19:36:33.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.443+0000 7f6ca6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:33.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.448+0000 7f6ca6ffd640 10 client.?.objecter ms_handle_connect 0x7f6cb00b1730 2024-03-25T19:36:33.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.448+0000 7f6ca6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:33.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.454+0000 7f6ca6ffd640 10 client.?.objecter ms_handle_connect 0x7f6cb00b4d60 2024-03-25T19:36:33.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.454+0000 7f6ca6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:33.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.456+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168840775 0 0) 0x7f6cb0143c30 con 0x7f6cb00b1730 2024-03-25T19:36:33.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.456+0000 7f6ca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:33.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.456+0000 7f6ca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:33.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.456+0000 7f6ca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:33.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.456+0000 7f6ca6ffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:36:33.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.456+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c90004eb0 con 0x7f6cb00b1730 2024-03-25T19:36:33.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.457+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583421541 0 0) 0x7f6cb0145840 con 0x7f6cb014be90 2024-03-25T19:36:33.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.457+0000 7f6ca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:33.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.457+0000 7f6ca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:33.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.457+0000 7f6ca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:33.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.457+0000 7f6ca6ffd640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:36:33.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.457+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb0143c30 con 0x7f6cb014be90 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3851206236 0 0) 0x7f6cb0143c30 con 0x7f6cb014be90 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb0145840 con 0x7f6cb014be90 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6ca8001c10 con 0x7f6cb014be90 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:33.477 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388536023 0 0) 0x7f6c90004eb0 con 0x7f6cb00b1730 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cb0143c30 con 0x7f6cb00b1730 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6ca0001b00 con 0x7f6cb00b1730 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:33.478 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.477+0000 7f6ca6ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:33.479 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:33.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.479+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350015290 0 0) 0x7f6cb0144150 con 0x7f6cb00b4d60 2024-03-25T19:36:33.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.479+0000 7f6ca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:33.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.479+0000 7f6ca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:33.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.479+0000 7f6ca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:33.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.479+0000 7f6ca6ffd640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:36:33.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.479+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c90004eb0 con 0x7f6cb00b4d60 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.493+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2190830279 0 0) 0x7f6cb0145840 con 0x7f6cb014be90 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 >> v1:172.21.15.143:6790/0 conn(0x7f6cb00b1730 legacy=0x7f6cb012fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 >> v1:172.21.15.143:6789/0 conn(0x7f6cb00b4d60 legacy=0x7f6cb01d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 1 monclient: found mon.b 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0143900 con 0x7f6cb014be90 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:33.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6ca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 5 monclient: authenticate success, global_id 4522 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 10 monclient: _renew_subs 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb0137990 con 0x7f6cb014be90 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 10 monclient: _renew_subs 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb01307e0 con 0x7f6cb014be90 2024-03-25T19:36:33.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.494+0000 7f6cb4e40640 1 librados: init done 2024-03-25T19:36:33.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.495+0000 7f6cb4e40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:33.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.495+0000 7f6cb4e40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:33.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.495+0000 7f6cb4e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:33.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.495+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb01401f0 con 0x7f6cb014be90 2024-03-25T19:36:33.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.779+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ca8003990 con 0x7f6cb014be90 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.779+0000 7f6ca6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.779+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6ca8002100 con 0x7f6cb014be90 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.779+0000 7f6ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.779+0000 7f6ca6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.779+0000 7f6ca6ffd640 10 monclient: dump: 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:33.780 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:33.781 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:33.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.779+0000 7f6cad59a640 4 set_mon_vals no callback set 2024-03-25T19:36:33.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.780+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f6ca8003e20 con 0x7f6cb014be90 2024-03-25T19:36:33.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.781+0000 7f6cae59c640 10 monclient: get_auth_request con 0x7f6c9006df00 auth_method 0 2024-03-25T19:36:33.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.781+0000 7f6ca6ffd640 10 client.4522.objecter ms_handle_connect 0x7f6c9006df00 2024-03-25T19:36:33.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.786+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f6ca8054940 con 0x7f6cb014be90 2024-03-25T19:36:33.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.786+0000 7f6ca6ffd640 10 client.4522.objecter ms_dispatch 0x7f6cb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:33.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.786+0000 7f6ca6ffd640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:33.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.786+0000 7f6ca6ffd640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:33.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.786+0000 7f6ca6ffd640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T19:36:33.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.786+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ca8050450 con 0x7f6cb014be90 2024-03-25T19:36:33.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.786+0000 7f6ca6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:33.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.786+0000 7f6ca6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:33.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.994+0000 7f6cb4e40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:33.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.994+0000 7f6cb4e40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:33.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.994+0000 7f6cb4e40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:33.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:33.994+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cb00627b0 con 0x7f6cb014be90 2024-03-25T19:36:34.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.083+0000 7f6ca6ffd640 1 -- 172.21.15.143:0/3954037275 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f6ca8052380 con 0x7f6cb014be90 2024-03-25T19:36:34.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.083+0000 7f6ca6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:34.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.083+0000 7f6ca6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:34.083 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:36:34.083 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:34.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 10 librados: watch_flush enter 2024-03-25T19:36:34.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 10 librados: watch_flush exit 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 10 client.4522.objecter successfully canceled tick 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 >> v1:172.21.15.143:6800/1292282666 conn(0x7f6c9006df00 legacy=0x7f6c900703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 10 monclient: shutdown 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 >> v1:172.21.15.203:6789/0 conn(0x7f6cb014be90 legacy=0x7f6cb01e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 shutdown_connections 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 >> 172.21.15.143:0/3954037275 conn(0x7f6cb00617a0 msgr2=0x7f6cb013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 shutdown_connections 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.086+0000 7f6cb4e40640 1 -- 172.21.15.143:0/3954037275 wait complete. 2024-03-25T19:36:34.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:34.087+0000 7f6cb4e40640 1 librados: shutdown 2024-03-25T19:36:34.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:39.017 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:05.581 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:39.022 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:05.587 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:39.064 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:05.629 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:39.078 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:05.614 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:39.078 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:05.643 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:39.082 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:05.617 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:39.082 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:05.647 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:39.117 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:05.653 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:39.123 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:05.688 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:40.904 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.020+0000 7fdb7654d640 10 monclient: get_monmap_and_config 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.020+0000 7fdb7654d640 10 monclient: build_initial_monmap 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.020+0000 7fdb7654d640 10 monclient: monmap: 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:36:41.021330+0000 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:36:41.021330+0000 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:41.020 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:41.021 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:36:41.021 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:36:41.021 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 1 Processor -- start 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 1 -- start start 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 10 monclient: init 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb700b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb700b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb7014be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb700af940 con 0x7fdb700b4d60 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.021+0000 7fdb7654d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb700b0a70 con 0x7fdb700b1730 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.022+0000 7fdb7654d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb700b0d50 con 0x7fdb7014be90 2024-03-25T19:36:41.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.022+0000 7fdb7654d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:41.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.022+0000 7fdb7654d640 10 monclient(hunting): authenticate will time out at 6417.365669s 2024-03-25T19:36:41.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.035+0000 7fdb6ffff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fdb700b4d60 0x7fdb7013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:35600/0 (socket says 172.21.15.143:35600) 2024-03-25T19:36:41.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.036+0000 7fdb6ffff640 1 -- 172.21.15.143:0/3788742567 learned_addr learned my addr 172.21.15.143:0/3788742567 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:41.051 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.052+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581010672 0 0) 0x7fdb700b0a70 con 0x7fdb700b1730 2024-03-25T19:36:41.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.052+0000 7fdb6effd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:41.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.052+0000 7fdb6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:41.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.052+0000 7fdb6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:41.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.052+0000 7fdb6effd640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:36:41.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.052+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb58003520 con 0x7fdb700b1730 2024-03-25T19:36:41.057 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.058+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941407744 0 0) 0x7fdb700af940 con 0x7fdb700b4d60 2024-03-25T19:36:41.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.058+0000 7fdb6effd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:41.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.058+0000 7fdb6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:41.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.058+0000 7fdb6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:41.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.058+0000 7fdb6effd640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:36:41.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.058+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb700b0a70 con 0x7fdb700b4d60 2024-03-25T19:36:41.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.060+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743779604 0 0) 0x7fdb700b0d50 con 0x7fdb7014be90 2024-03-25T19:36:41.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.060+0000 7fdb6effd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:41.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.060+0000 7fdb6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:41.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.060+0000 7fdb6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:41.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.060+0000 7fdb6effd640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:36:41.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.060+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb700af940 con 0x7fdb7014be90 2024-03-25T19:36:41.065 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1990017066 0 0) 0x7fdb58003520 con 0x7fdb700b1730 2024-03-25T19:36:41.065 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 >> v1:172.21.15.143:6790/0 conn(0x7fdb7014be90 legacy=0x7fdb7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:41.065 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 >> v1:172.21.15.143:6789/0 conn(0x7fdb700b4d60 legacy=0x7fdb7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 1 monclient: found mon.noname-c 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb700b1030 con 0x7fdb700b1730 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 10 monclient: _finish_auth 0 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.065+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdb64001690 con 0x7fdb700b1730 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.066+0000 7fdb6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.066+0000 7fdb6effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:41.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.066+0000 7fdb6effd640 10 monclient: dump: 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.066+0000 7fdb7654d640 5 monclient: authenticate success, global_id 4525 2024-03-25T19:36:41.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.066+0000 7fdb7654d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.085+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb640017f0 con 0x7fdb700b1730 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.085+0000 7fdb6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.085+0000 7fdb6effd640 1 -- 172.21.15.143:0/3788742567 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdb64001c00 con 0x7fdb700b1730 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.085+0000 7fdb6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.085+0000 7fdb6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.085+0000 7fdb6effd640 10 monclient: dump: 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:41.085 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:41.086 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:41.086 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:41.086 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:41.086 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:41.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.085+0000 7fdb7654d640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:41.086 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.085+0000 7fdb7654d640 4 set_mon_vals no callback set 2024-03-25T19:36:41.088 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.086+0000 7fdb7654d640 10 monclient: shutdown 2024-03-25T19:36:41.088 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.086+0000 7fdb7654d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:41.088 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.086+0000 7fdb7654d640 1 -- 172.21.15.143:0/3788742567 >> v1:172.21.15.203:6789/0 conn(0x7fdb700b1730 legacy=0x7fdb700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:41.088 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.086+0000 7fdb7654d640 1 -- 172.21.15.143:0/3788742567 shutdown_connections 2024-03-25T19:36:41.088 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.086+0000 7fdb7654d640 1 -- 172.21.15.143:0/3788742567 >> 172.21.15.143:0/3788742567 conn(0x7fdb700617a0 msgr2=0x7fdb70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.086+0000 7fdb7654d640 1 -- 172.21.15.143:0/3788742567 shutdown_connections 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.086+0000 7fdb7654d640 1 -- 172.21.15.143:0/3788742567 wait complete. 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 10 monclient: build_initial_monmap 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 10 monclient: monmap: 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:36:41.089 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 1 librados: starting msgr at 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 1 librados: starting objecter 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 1 Processor -- start 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 1 -- start start 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 1 librados: setting wanted keys 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 1 librados: calling monclient init 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.087+0000 7fdb7654d640 10 monclient: init 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb700b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb700b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb7014be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb70142050 con 0x7fdb7014be90 2024-03-25T19:36:41.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb701dac50 con 0x7fdb700b4d60 2024-03-25T19:36:41.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb70140d00 con 0x7fdb700b1730 2024-03-25T19:36:41.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:41.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.088+0000 7fdb7654d640 10 monclient(hunting): authenticate will time out at 6417.432182s 2024-03-25T19:36:41.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.097+0000 7fdb6f7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fdb700b4d60 0x7fdb701db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:49964/0 (socket says 172.21.15.143:49964) 2024-03-25T19:36:41.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.097+0000 7fdb6f7fe640 1 -- 172.21.15.143:0/3833808711 learned_addr learned my addr 172.21.15.143:0/3833808711 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:41.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.100+0000 7fdb6cff9640 10 client.?.objecter ms_handle_connect 0x7fdb700b4d60 2024-03-25T19:36:41.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.100+0000 7fdb6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:41.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.107+0000 7fdb6cff9640 10 client.?.objecter ms_handle_connect 0x7fdb700b1730 2024-03-25T19:36:41.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.107+0000 7fdb6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:41.109 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.110+0000 7fdb6cff9640 10 client.?.objecter ms_handle_connect 0x7fdb7014be90 2024-03-25T19:36:41.109 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.110+0000 7fdb6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:41.114 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.115+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692513380 0 0) 0x7fdb701dac50 con 0x7fdb700b4d60 2024-03-25T19:36:41.115 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.115+0000 7fdb6cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:41.115 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.115+0000 7fdb6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:41.115 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.115+0000 7fdb6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:41.115 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.115+0000 7fdb6cff9640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:36:41.115 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.115+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5c004eb0 con 0x7fdb700b4d60 2024-03-25T19:36:41.116 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482367022 0 0) 0x7fdb70142050 con 0x7fdb7014be90 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb701dac50 con 0x7fdb7014be90 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48967757 0 0) 0x7fdb70140d00 con 0x7fdb700b1730 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:36:41.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.117+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb70142050 con 0x7fdb700b1730 2024-03-25T19:36:41.129 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.129+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3778257018 0 0) 0x7fdb70142050 con 0x7fdb700b1730 2024-03-25T19:36:41.129 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.129+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb70140d00 con 0x7fdb700b1730 2024-03-25T19:36:41.129 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.129+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdb600018f0 con 0x7fdb700b1730 2024-03-25T19:36:41.129 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.129+0000 7fdb6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:41.129 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.129+0000 7fdb6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:41.129 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.129+0000 7fdb6cff9640 10 monclient(hunting): dump: 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:41.130 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.129+0000 7fdb6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:41.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.135+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2833213369 0 0) 0x7fdb5c004eb0 con 0x7fdb700b4d60 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.135+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb70142050 con 0x7fdb700b4d60 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.135+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdb64001bc0 con 0x7fdb700b4d60 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.135+0000 7fdb6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.135+0000 7fdb6cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.135+0000 7fdb6cff9640 10 monclient(hunting): dump: 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:41.135 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:41.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3935495460 0 0) 0x7fdb70142050 con 0x7fdb700b4d60 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 >> v1:172.21.15.143:6790/0 conn(0x7fdb700b1730 legacy=0x7fdb701405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 >> v1:172.21.15.143:6789/0 conn(0x7fdb7014be90 legacy=0x7fdb701e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 1 monclient: found mon.b 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb701423e0 con 0x7fdb700b4d60 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb7654d640 5 monclient: authenticate success, global_id 4528 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb7654d640 10 monclient: _renew_subs 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb7654d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:41.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.146+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7013b470 con 0x7fdb700b4d60 2024-03-25T19:36:41.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.147+0000 7fdb7654d640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:41.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.147+0000 7fdb7654d640 10 monclient: _renew_subs 2024-03-25T19:36:41.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.147+0000 7fdb7654d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:41.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.147+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb701dba40 con 0x7fdb700b4d60 2024-03-25T19:36:41.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.147+0000 7fdb7654d640 1 librados: init done 2024-03-25T19:36:41.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.148+0000 7fdb7654d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:41.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.148+0000 7fdb7654d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:41.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.148+0000 7fdb7654d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:41.149 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.148+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb701401f0 con 0x7fdb700b4d60 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.434+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb640011f0 con 0x7fdb700b4d60 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.434+0000 7fdb6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.434+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdb64003b00 con 0x7fdb700b4d60 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.434+0000 7fdb6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.434+0000 7fdb6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.434+0000 7fdb6cff9640 10 monclient: dump: 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:41.434 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:41.435 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:41.435 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:41.435 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:41.435 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:41.435 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:41.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.434+0000 7fdb6e7fc640 4 set_mon_vals no callback set 2024-03-25T19:36:41.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.435+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fdb64003d20 con 0x7fdb700b4d60 2024-03-25T19:36:41.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.435+0000 7fdb6ffff640 10 monclient: get_auth_request con 0x7fdb5c06dfd0 auth_method 0 2024-03-25T19:36:41.436 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.436+0000 7fdb6cff9640 10 client.4528.objecter ms_handle_connect 0x7fdb5c06dfd0 2024-03-25T19:36:41.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.442+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fdb64054880 con 0x7fdb700b4d60 2024-03-25T19:36:41.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.442+0000 7fdb6cff9640 10 client.4528.objecter ms_dispatch 0x7fdb70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:41.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.442+0000 7fdb6cff9640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:41.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.442+0000 7fdb6cff9640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:41.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.442+0000 7fdb6cff9640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T19:36:41.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.442+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb64050390 con 0x7fdb700b4d60 2024-03-25T19:36:41.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.442+0000 7fdb6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:41.442 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.442+0000 7fdb6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:41.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.663+0000 7fdb7654d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:41.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.663+0000 7fdb7654d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:41.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.663+0000 7fdb7654d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:41.663 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.663+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb70140e90 con 0x7fdb700b4d60 2024-03-25T19:36:41.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.690+0000 7fdb6cff9640 1 -- 172.21.15.143:0/3833808711 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fdb640522c0 con 0x7fdb700b4d60 2024-03-25T19:36:41.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.690+0000 7fdb6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:41.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.690+0000 7fdb6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:41.690 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:36:41.691 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.692+0000 7fdb7654d640 10 librados: watch_flush enter 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.692+0000 7fdb7654d640 10 librados: watch_flush exit 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.692+0000 7fdb7654d640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.692+0000 7fdb7654d640 10 client.4528.objecter successfully canceled tick 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 >> v1:172.21.15.143:6800/1292282666 conn(0x7fdb5c06dfd0 legacy=0x7fdb5c070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 10 monclient: shutdown 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 >> v1:172.21.15.203:6789/0 conn(0x7fdb700b4d60 legacy=0x7fdb701db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 shutdown_connections 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 >> 172.21.15.143:0/3833808711 conn(0x7fdb700617a0 msgr2=0x7fdb700b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 shutdown_connections 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 1 -- 172.21.15.143:0/3833808711 wait complete. 2024-03-25T19:36:41.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:41.693+0000 7fdb7654d640 1 librados: shutdown 2024-03-25T19:36:41.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:48.508 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.624+0000 7f373cece640 10 monclient: get_monmap_and_config 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.624+0000 7f373cece640 10 monclient: build_initial_monmap 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.624+0000 7f373cece640 10 monclient: monmap: 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:36:48.625662+0000 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:36:48.625662+0000 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:48.624 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:48.625 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:36:48.625 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:36:48.625 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:48.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.625+0000 7f373cece640 1 Processor -- start 2024-03-25T19:36:48.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.625+0000 7f373cece640 1 -- start start 2024-03-25T19:36:48.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.625+0000 7f373cece640 10 monclient: init 2024-03-25T19:36:48.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:48.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:36:48.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 10 monclient(hunting): picked mon.noname-a con 0x7f37380a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:48.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 10 monclient(hunting): picked mon.noname-c con 0x7f373813f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:48.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 10 monclient(hunting): picked mon.noname-b con 0x7f37381430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:48.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3738142520 con 0x7f37380a98f0 2024-03-25T19:36:48.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37380b28a0 con 0x7f373813f710 2024-03-25T19:36:48.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37380b2a30 con 0x7f37381430c0 2024-03-25T19:36:48.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:48.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.626+0000 7f373cece640 10 monclient(hunting): authenticate will time out at 6424.970174s 2024-03-25T19:36:48.640 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.640+0000 7f3735d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f373813f710 0x7f37381429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:54444/0 (socket says 172.21.15.143:54444) 2024-03-25T19:36:48.640 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.640+0000 7f3735d9b640 1 -- 172.21.15.143:0/2528571557 learned_addr learned my addr 172.21.15.143:0/2528571557 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:48.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.657+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273006094 0 0) 0x7f37380b28a0 con 0x7f373813f710 2024-03-25T19:36:48.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.657+0000 7f373559a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:48.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.657+0000 7f373559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:48.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.657+0000 7f373559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:48.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.657+0000 7f373559a640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:36:48.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.657+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3720003520 con 0x7f373813f710 2024-03-25T19:36:48.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.659+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891678831 0 0) 0x7f3738142520 con 0x7f37380a98f0 2024-03-25T19:36:48.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.659+0000 7f373559a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:48.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.659+0000 7f373559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:48.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.659+0000 7f373559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:48.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.659+0000 7f373559a640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:36:48.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.659+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37380b28a0 con 0x7f37380a98f0 2024-03-25T19:36:48.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.664+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298230421 0 0) 0x7f37380b2a30 con 0x7f37381430c0 2024-03-25T19:36:48.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.664+0000 7f373559a640 10 monclient(hunting): _negotiate 2024-03-25T19:36:48.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.664+0000 7f373559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:48.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.664+0000 7f373559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:48.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.664+0000 7f373559a640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:36:48.664 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.664+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3738142520 con 0x7f37381430c0 2024-03-25T19:36:48.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1781366669 0 0) 0x7f3720003520 con 0x7f373813f710 2024-03-25T19:36:48.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 >> v1:172.21.15.143:6790/0 conn(0x7f37381430c0 legacy=0x7f373814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:48.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 >> v1:172.21.15.143:6789/0 conn(0x7f37380a98f0 legacy=0x7f37380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:48.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 10 monclient: _finish_hunting 0 2024-03-25T19:36:48.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 1 monclient: found mon.noname-c 2024-03-25T19:36:48.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:48.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37380b44c0 con 0x7f373813f710 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 10 monclient: _finish_auth 0 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 10 monclient: _check_auth_tickets 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f372c001690 con 0x7f373813f710 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373cece640 5 monclient: authenticate success, global_id 4531 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373cece640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373559a640 10 monclient: dump: 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:48.676 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:48.677 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:48.677 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:48.677 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:48.677 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:48.677 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:48.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.675+0000 7f373cece640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:48.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.691+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f372c0017f0 con 0x7f373813f710 2024-03-25T19:36:48.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.691+0000 7f373559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:48.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.691+0000 7f373559a640 1 -- 172.21.15.143:0/2528571557 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f372c001c00 con 0x7f373813f710 2024-03-25T19:36:48.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.691+0000 7f373cece640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:48.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.691+0000 7f373cece640 4 set_mon_vals no callback set 2024-03-25T19:36:48.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.691+0000 7f373559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:48.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.691+0000 7f373559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.691+0000 7f373559a640 10 monclient: dump: 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:48.692 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:48.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.692+0000 7f373cece640 10 monclient: shutdown 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.692+0000 7f373cece640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.692+0000 7f373cece640 1 -- 172.21.15.143:0/2528571557 >> v1:172.21.15.203:6789/0 conn(0x7f373813f710 legacy=0x7f37381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.692+0000 7f373cece640 1 -- 172.21.15.143:0/2528571557 shutdown_connections 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.692+0000 7f373cece640 1 -- 172.21.15.143:0/2528571557 >> 172.21.15.143:0/2528571557 conn(0x7f37380b7ea0 msgr2=0x7f37380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.692+0000 7f373cece640 1 -- 172.21.15.143:0/2528571557 shutdown_connections 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.692+0000 7f373cece640 1 -- 172.21.15.143:0/2528571557 wait complete. 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.693+0000 7f373cece640 10 monclient: build_initial_monmap 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.693+0000 7f373cece640 10 monclient: monmap: 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:36:48.694 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.694+0000 7f373cece640 1 librados: starting msgr at 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.694+0000 7f373cece640 1 librados: starting objecter 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.694+0000 7f373cece640 1 Processor -- start 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.694+0000 7f373cece640 1 -- start start 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.694+0000 7f373cece640 1 librados: setting wanted keys 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.694+0000 7f373cece640 1 librados: calling monclient init 2024-03-25T19:36:48.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.694+0000 7f373cece640 10 monclient: init 2024-03-25T19:36:48.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:48.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:36:48.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 10 monclient(hunting): picked mon.noname-b con 0x7f37380a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:48.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 10 monclient(hunting): picked mon.noname-c con 0x7f373813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:48.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 10 monclient(hunting): picked mon.noname-a con 0x7f37381430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:48.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37381d7750 con 0x7f37381430c0 2024-03-25T19:36:48.696 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37381d85d0 con 0x7f37380a98f0 2024-03-25T19:36:48.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37381d79a0 con 0x7f373813f710 2024-03-25T19:36:48.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:48.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.695+0000 7f373cece640 10 monclient(hunting): authenticate will time out at 6425.038866s 2024-03-25T19:36:48.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.697+0000 7f3735d9b640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f373813f710 0x7f37381d4990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:52900/0 (socket says 172.21.15.143:52900) 2024-03-25T19:36:48.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.697+0000 7f3735d9b640 1 -- 172.21.15.143:0/948559235 learned_addr learned my addr 172.21.15.143:0/948559235 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:48.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.698+0000 7f3726ffd640 10 client.?.objecter ms_handle_connect 0x7f373813f710 2024-03-25T19:36:48.697 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.698+0000 7f3726ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:48.703 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 10 client.?.objecter ms_handle_connect 0x7f37380a98f0 2024-03-25T19:36:48.703 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:48.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104126808 0 0) 0x7f37381d79a0 con 0x7f373813f710 2024-03-25T19:36:48.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:48.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:48.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:48.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:36:48.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3718004eb0 con 0x7f373813f710 2024-03-25T19:36:48.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 10 client.?.objecter ms_handle_connect 0x7f37381430c0 2024-03-25T19:36:48.704 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.704+0000 7f3726ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:48.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.718+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645627984 0 0) 0x7f37381d85d0 con 0x7f37380a98f0 2024-03-25T19:36:48.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.718+0000 7f3726ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:48.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.718+0000 7f3726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:48.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.718+0000 7f3726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:48.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.718+0000 7f3726ffd640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:36:48.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.718+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37381d79a0 con 0x7f37380a98f0 2024-03-25T19:36:48.723 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.723+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2659583303 0 0) 0x7f3718004eb0 con 0x7f373813f710 2024-03-25T19:36:48.723 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.724+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37381d85d0 con 0x7f373813f710 2024-03-25T19:36:48.723 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.724+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f372c001b40 con 0x7f373813f710 2024-03-25T19:36:48.723 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.724+0000 7f3726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.724+0000 7f3726ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.724+0000 7f3726ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:48.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.724+0000 7f3726ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:48.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.726+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838307188 0 0) 0x7f37381d7750 con 0x7f37381430c0 2024-03-25T19:36:48.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.726+0000 7f3726ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:48.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.726+0000 7f3726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:48.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.726+0000 7f3726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:48.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.726+0000 7f3726ffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:36:48.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.726+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3718004eb0 con 0x7f37381430c0 2024-03-25T19:36:48.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.732+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769898658 0 0) 0x7f37381d79a0 con 0x7f37380a98f0 2024-03-25T19:36:48.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.732+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37381d7750 con 0x7f37380a98f0 2024-03-25T19:36:48.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.732+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f3728001970 con 0x7f37380a98f0 2024-03-25T19:36:48.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.732+0000 7f3726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:48.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.732+0000 7f3726ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:48.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.732+0000 7f3726ffd640 10 monclient(hunting): dump: 2024-03-25T19:36:48.732 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:48.732 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:48.733 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:48.733 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:48.733 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:48.733 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:48.733 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:48.733 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:48.733 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:48.733 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2219712594 0 0) 0x7f37381d85d0 con 0x7f373813f710 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 >> v1:172.21.15.203:6789/0 conn(0x7f37380a98f0 legacy=0x7f37381d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 >> v1:172.21.15.143:6789/0 conn(0x7f37381430c0 legacy=0x7f3738138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 1 monclient: found mon.c 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3738137d60 con 0x7f373813f710 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:48.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f3726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 10 monclient: _renew_subs 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37381d50a0 con 0x7f373813f710 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 10 monclient: _renew_subs 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37381d43c0 con 0x7f373813f710 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.734+0000 7f373cece640 1 librados: init done 2024-03-25T19:36:48.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.735+0000 7f373cece640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:48.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.735+0000 7f373cece640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:48.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.735+0000 7f373cece640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:36:48.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:48.735+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3738136b00 con 0x7f373813f710 2024-03-25T19:36:49.039 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.039+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f372c004a90 con 0x7f373813f710 2024-03-25T19:36:49.039 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.039+0000 7f3726ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:49.039 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.039+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f372c004c00 con 0x7f373813f710 2024-03-25T19:36:49.039 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.039+0000 7f3726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:49.039 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.039+0000 7f3726ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.039+0000 7f3726ffd640 10 monclient: dump: 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:49.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.039+0000 7f3734d99640 4 set_mon_vals no callback set 2024-03-25T19:36:49.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.040+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f372c01abd0 con 0x7f373813f710 2024-03-25T19:36:49.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.041+0000 7f373659c640 10 monclient: get_auth_request con 0x7f371806df70 auth_method 0 2024-03-25T19:36:49.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.041+0000 7f3726ffd640 10 client.4526.objecter ms_handle_connect 0x7f371806df70 2024-03-25T19:36:49.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.048+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f372c0547b0 con 0x7f373813f710 2024-03-25T19:36:49.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.048+0000 7f3726ffd640 10 client.4526.objecter ms_dispatch 0x7f3738000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:49.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.048+0000 7f3726ffd640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:49.049 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.048+0000 7f3726ffd640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:49.049 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.048+0000 7f3726ffd640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-25T19:36:49.049 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.048+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f372c0500f0 con 0x7f373813f710 2024-03-25T19:36:49.049 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.048+0000 7f3726ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:49.049 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.048+0000 7f3726ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:49.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.283+0000 7f373cece640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:49.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.283+0000 7f373cece640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:49.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.283+0000 7f373cece640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:36:49.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.283+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37381d7b30 con 0x7f373813f710 2024-03-25T19:36:49.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.304+0000 7f3726ffd640 1 -- 172.21.15.143:0/948559235 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f372c0521f0 con 0x7f373813f710 2024-03-25T19:36:49.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.304+0000 7f3726ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:49.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.304+0000 7f3726ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:49.304 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:36:49.304 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.306+0000 7f373cece640 10 librados: watch_flush enter 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.306+0000 7f373cece640 10 librados: watch_flush exit 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.306+0000 7f373cece640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.306+0000 7f373cece640 10 client.4526.objecter successfully canceled tick 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.306+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 >> v1:172.21.15.143:6800/1292282666 conn(0x7f371806df70 legacy=0x7f3718070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.306+0000 7f373cece640 10 monclient: shutdown 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.306+0000 7f373cece640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.306+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 >> v1:172.21.15.143:6790/0 conn(0x7f373813f710 legacy=0x7f37381d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.307+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 shutdown_connections 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.307+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 >> 172.21.15.143:0/948559235 conn(0x7f37380b7ea0 msgr2=0x7f3738074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:49.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.307+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 shutdown_connections 2024-03-25T19:36:49.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.307+0000 7f373cece640 1 -- 172.21.15.143:0/948559235 wait complete. 2024-03-25T19:36:49.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:49.307+0000 7f373cece640 1 librados: shutdown 2024-03-25T19:36:49.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:56.125 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:36:56.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.240+0000 7fbbfe8c9640 10 monclient: get_monmap_and_config 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.240+0000 7fbbfe8c9640 10 monclient: build_initial_monmap 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.241+0000 7fbbfe8c9640 10 monclient: monmap: 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:36:56.241902+0000 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:36:56.241902+0000 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:36:56.241 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:56.242 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.242+0000 7fbbfe8c9640 1 Processor -- start 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.242+0000 7fbbfe8c9640 1 -- start start 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.242+0000 7fbbfe8c9640 10 monclient: init 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.242+0000 7fbbfe8c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.242+0000 7fbbfe8c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.242+0000 7fbbfe8c9640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.242+0000 7fbbfe8c9640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.243+0000 7fbbfe8c9640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf81430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.243+0000 7fbbfe8c9640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf8142520 con 0x7fbbf813f710 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.243+0000 7fbbfe8c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf80b28a0 con 0x7fbbf80a98f0 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.243+0000 7fbbfe8c9640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf80b2a30 con 0x7fbbf81430c0 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.243+0000 7fbbfe8c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:56.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.243+0000 7fbbfe8c9640 10 monclient(hunting): authenticate will time out at 6432.586893s 2024-03-25T19:36:56.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.253+0000 7fbbf7fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fbbf80a98f0 0x7fbbf80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:59292/0 (socket says 172.21.15.143:59292) 2024-03-25T19:36:56.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.253+0000 7fbbf7fff640 1 -- 172.21.15.143:0/4217169223 learned_addr learned my addr 172.21.15.143:0/4217169223 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:56.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.269+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038137101 0 0) 0x7fbbf80b2a30 con 0x7fbbf81430c0 2024-03-25T19:36:56.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.269+0000 7fbbf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:56.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.269+0000 7fbbf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:56.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.269+0000 7fbbf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:56.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.269+0000 7fbbf6ffd640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:36:56.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.269+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe0003520 con 0x7fbbf81430c0 2024-03-25T19:36:56.275 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.275+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445696322 0 0) 0x7fbbf80b28a0 con 0x7fbbf80a98f0 2024-03-25T19:36:56.275 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.275+0000 7fbbf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:56.275 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.275+0000 7fbbf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:56.275 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.275+0000 7fbbf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:56.275 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.275+0000 7fbbf6ffd640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:36:56.275 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.275+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf80b2a30 con 0x7fbbf80a98f0 2024-03-25T19:36:56.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.281+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086474261 0 0) 0x7fbbf8142520 con 0x7fbbf813f710 2024-03-25T19:36:56.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.281+0000 7fbbf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:36:56.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.281+0000 7fbbf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:56.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.281+0000 7fbbf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:56.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.281+0000 7fbbf6ffd640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:36:56.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.281+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf80b28a0 con 0x7fbbf813f710 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2311910105 0 0) 0x7fbbe0003520 con 0x7fbbf81430c0 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 >> v1:172.21.15.203:6789/0 conn(0x7fbbf80a98f0 legacy=0x7fbbf80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 >> v1:172.21.15.143:6789/0 conn(0x7fbbf813f710 legacy=0x7fbbf81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 1 monclient: found mon.noname-b 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf80b44c0 con 0x7fbbf81430c0 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:56.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbbf0001a10 con 0x7fbbf81430c0 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbf6ffd640 10 monclient: dump: 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbfe8c9640 5 monclient: authenticate success, global_id 4529 2024-03-25T19:36:56.288 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.286+0000 7fbbfe8c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:36:56.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf0001b70 con 0x7fbbf81430c0 2024-03-25T19:36:56.297 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbf6ffd640 1 -- 172.21.15.143:0/4217169223 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbbf0001f80 con 0x7fbbf81430c0 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbfe8c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbfe8c9640 4 set_mon_vals no callback set 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbf6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbf6ffd640 10 monclient: dump: 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:56.298 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:56.299 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:56.299 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:56.299 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbfe8c9640 10 monclient: shutdown 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbfe8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.298+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/4217169223 >> v1:172.21.15.143:6790/0 conn(0x7fbbf81430c0 legacy=0x7fbbf814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.299+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/4217169223 shutdown_connections 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.299+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/4217169223 >> 172.21.15.143:0/4217169223 conn(0x7fbbf80b7ea0 msgr2=0x7fbbf80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.299+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/4217169223 shutdown_connections 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.299+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/4217169223 wait complete. 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.299+0000 7fbbfe8c9640 10 monclient: build_initial_monmap 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.299+0000 7fbbfe8c9640 10 monclient: monmap: 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:36:56.301 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.300+0000 7fbbfe8c9640 1 librados: starting msgr at 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.300+0000 7fbbfe8c9640 1 librados: starting objecter 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.300+0000 7fbbfe8c9640 1 Processor -- start 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.300+0000 7fbbfe8c9640 1 -- start start 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.300+0000 7fbbfe8c9640 1 librados: setting wanted keys 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.300+0000 7fbbfe8c9640 1 librados: calling monclient init 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.300+0000 7fbbfe8c9640 10 monclient: init 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:36:56.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:36:56.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf80a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:36:56.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:36:56.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:36:56.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf8138b40 con 0x7fbbf80a98f0 2024-03-25T19:36:56.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf8138e20 con 0x7fbbf81430c0 2024-03-25T19:36:56.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf8139100 con 0x7fbbf813f710 2024-03-25T19:36:56.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:36:56.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.301+0000 7fbbfe8c9640 10 monclient(hunting): authenticate will time out at 6432.645064s 2024-03-25T19:36:56.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.306+0000 7fbbf7fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fbbf80a98f0 0x7fbbf8137380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:38880/0 (socket says 172.21.15.143:38880) 2024-03-25T19:36:56.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.306+0000 7fbbf7fff640 1 -- 172.21.15.143:0/2591273334 learned_addr learned my addr 172.21.15.143:0/2591273334 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:36:56.314 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.314+0000 7fbbf4ff9640 10 client.?.objecter ms_handle_connect 0x7fbbf81430c0 2024-03-25T19:36:56.314 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.314+0000 7fbbf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:56.315 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.315+0000 7fbbf4ff9640 10 client.?.objecter ms_handle_connect 0x7fbbf80a98f0 2024-03-25T19:36:56.315 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.315+0000 7fbbf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:56.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.326+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755091015 0 0) 0x7fbbf8138e20 con 0x7fbbf81430c0 2024-03-25T19:36:56.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.326+0000 7fbbf4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:56.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.326+0000 7fbbf4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:56.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.326+0000 7fbbf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:56.326 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.326+0000 7fbbf4ff9640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:36:56.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.326+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbe4004eb0 con 0x7fbbf81430c0 2024-03-25T19:36:56.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.329+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734817417 0 0) 0x7fbbf8138b40 con 0x7fbbf80a98f0 2024-03-25T19:36:56.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.329+0000 7fbbf4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:36:56.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.329+0000 7fbbf4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:36:56.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.329+0000 7fbbf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:36:56.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.329+0000 7fbbf4ff9640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:36:56.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.329+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf8138e20 con 0x7fbbf80a98f0 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.337+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644207197 0 0) 0x7fbbf8138e20 con 0x7fbbf80a98f0 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.337+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbf8138b40 con 0x7fbbf80a98f0 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.337+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbbe80019b0 con 0x7fbbf80a98f0 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.337+0000 7fbbf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.337+0000 7fbbf4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.337+0000 7fbbf4ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:56.337 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:56.338 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:56.338 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:56.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.337+0000 7fbbf4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:36:56.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.338+0000 7fbbf4ff9640 10 client.?.objecter ms_handle_connect 0x7fbbf813f710 2024-03-25T19:36:56.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.338+0000 7fbbf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:36:56.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.346+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 90593436 0 0) 0x7fbbe4004eb0 con 0x7fbbf81430c0 2024-03-25T19:36:56.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.346+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbf8138e20 con 0x7fbbf81430c0 2024-03-25T19:36:56.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.346+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbbf0001e80 con 0x7fbbf81430c0 2024-03-25T19:36:56.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.346+0000 7fbbf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:56.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.346+0000 7fbbf4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:56.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.346+0000 7fbbf4ff9640 10 monclient(hunting): dump: 2024-03-25T19:36:56.346 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:56.346 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:56.347 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.347 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.347 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:56.347 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:56.347 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:56.347 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:56.347 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:56.347 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:56.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.355+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2243300011 0 0) 0x7fbbf8138e20 con 0x7fbbf81430c0 2024-03-25T19:36:56.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 >> v1:172.21.15.143:6790/0 conn(0x7fbbf813f710 legacy=0x7fbbf8137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 >> v1:172.21.15.143:6789/0 conn(0x7fbbf80a98f0 legacy=0x7fbbf8137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 1 monclient: found mon.b 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf81397a0 con 0x7fbbf81430c0 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbf4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 5 monclient: authenticate success, global_id 4540 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 10 monclient: _renew_subs 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf8136aa0 con 0x7fbbf81430c0 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:36:56.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 10 monclient: _renew_subs 2024-03-25T19:36:56.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:56.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf813c320 con 0x7fbbf81430c0 2024-03-25T19:36:56.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.356+0000 7fbbfe8c9640 1 librados: init done 2024-03-25T19:36:56.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.357+0000 7fbbfe8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:56.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.357+0000 7fbbfe8c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:56.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.357+0000 7fbbfe8c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:56.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.357+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf81395b0 con 0x7fbbf81430c0 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.628+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbf0002020 con 0x7fbbf81430c0 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.628+0000 7fbbf4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.628+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fbbf0003dc0 con 0x7fbbf81430c0 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.628+0000 7fbbf4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.628+0000 7fbbf4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.628+0000 7fbbf4ff9640 10 monclient: dump: 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:36:56.628 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:36:56.629 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:36:56.629 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:36:56.629 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:36:56.629 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:36:56.629 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:36:56.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.628+0000 7fbbf67fc640 4 set_mon_vals no callback set 2024-03-25T19:36:56.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.629+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fbbf0019d90 con 0x7fbbf81430c0 2024-03-25T19:36:56.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.629+0000 7fbbf7fff640 10 monclient: get_auth_request con 0x7fbbe406dee0 auth_method 0 2024-03-25T19:36:56.630 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.630+0000 7fbbf4ff9640 10 client.4540.objecter ms_handle_connect 0x7fbbe406dee0 2024-03-25T19:36:56.632 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.633+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fbbf00538f0 con 0x7fbbf81430c0 2024-03-25T19:36:56.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.633+0000 7fbbf4ff9640 10 client.4540.objecter ms_dispatch 0x7fbbf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:36:56.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.633+0000 7fbbf4ff9640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:36:56.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.633+0000 7fbbf4ff9640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:36:56.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.633+0000 7fbbf4ff9640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T19:36:56.636 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.636+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbf004f400 con 0x7fbbf81430c0 2024-03-25T19:36:56.636 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.637+0000 7fbbf4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:36:56.636 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.637+0000 7fbbf4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:36:56.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.875+0000 7fbbfe8c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:56.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.875+0000 7fbbfe8c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:56.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.875+0000 7fbbfe8c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:36:56.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.875+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbf8139930 con 0x7fbbf81430c0 2024-03-25T19:36:56.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.900+0000 7fbbf4ff9640 1 -- 172.21.15.143:0/2591273334 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fbbf0051330 con 0x7fbbf81430c0 2024-03-25T19:36:56.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.900+0000 7fbbf4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:36:56.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.900+0000 7fbbf4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:36:56.900 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:36:56.900 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.902+0000 7fbbfe8c9640 10 librados: watch_flush enter 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.902+0000 7fbbfe8c9640 10 librados: watch_flush exit 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.902+0000 7fbbfe8c9640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.902+0000 7fbbfe8c9640 10 client.4540.objecter successfully canceled tick 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 >> v1:172.21.15.143:6800/1292282666 conn(0x7fbbe406dee0 legacy=0x7fbbe40703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 10 monclient: shutdown 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 >> v1:172.21.15.203:6789/0 conn(0x7fbbf81430c0 legacy=0x7fbbf81e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 shutdown_connections 2024-03-25T19:36:56.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 >> 172.21.15.143:0/2591273334 conn(0x7fbbf80b7ea0 msgr2=0x7fbbf814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:36:56.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 shutdown_connections 2024-03-25T19:36:56.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 1 -- 172.21.15.143:0/2591273334 wait complete. 2024-03-25T19:36:56.904 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:36:56.903+0000 7fbbfe8c9640 1 librados: shutdown 2024-03-25T19:36:56.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:36:59.142 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:25.707 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:59.151 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:25.716 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:59.182 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:25.746 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:59.195 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:25.760 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:59.202 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:25.767 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:59.209 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:25.745 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:59.213 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:25.749 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:59.237 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:25.802 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:36:59.245 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:25.781 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:03.720 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:03.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.834+0000 7fe59d2ef640 10 monclient: get_monmap_and_config 2024-03-25T19:37:03.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.834+0000 7fe59d2ef640 10 monclient: build_initial_monmap 2024-03-25T19:37:03.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.834+0000 7fe59d2ef640 10 monclient: monmap: 2024-03-25T19:37:03.833 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:03.833 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:03.834 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:37:03.834933+0000 2024-03-25T19:37:03.834 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:37:03.834933+0000 2024-03-25T19:37:03.834 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:03.834 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:03.834 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:03.834 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:37:03.834 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:37:03.834 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:03.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.834+0000 7fe59d2ef640 1 Processor -- start 2024-03-25T19:37:03.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.834+0000 7fe59d2ef640 1 -- start start 2024-03-25T19:37:03.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.834+0000 7fe59d2ef640 10 monclient: init 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.835+0000 7fe59d2ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.835+0000 7fe59d2ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.835+0000 7fe59d2ef640 10 monclient(hunting): picked mon.noname-c con 0x7fe5980b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.835+0000 7fe59d2ef640 10 monclient(hunting): picked mon.noname-b con 0x7fe5980b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.835+0000 7fe59d2ef640 10 monclient(hunting): picked mon.noname-a con 0x7fe59814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.835+0000 7fe59d2ef640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5980af940 con 0x7fe59814be90 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.835+0000 7fe59d2ef640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5980b0a70 con 0x7fe5980b4d60 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.836+0000 7fe59d2ef640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5980b0d50 con 0x7fe5980b1730 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.836+0000 7fe59d2ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:03.836 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.836+0000 7fe59d2ef640 10 monclient(hunting): authenticate will time out at 6440.179686s 2024-03-25T19:37:03.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.857+0000 7fe596ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe5980b4d60 0x7fe59813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:53746/0 (socket says 172.21.15.143:53746) 2024-03-25T19:37:03.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.857+0000 7fe596ffd640 1 -- 172.21.15.143:0/823145294 learned_addr learned my addr 172.21.15.143:0/823145294 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:03.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.875+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075104889 0 0) 0x7fe5980b0a70 con 0x7fe5980b4d60 2024-03-25T19:37:03.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.875+0000 7fe595ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:03.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.875+0000 7fe595ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:03.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.875+0000 7fe595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:03.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.875+0000 7fe595ffb640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:37:03.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.875+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe580003520 con 0x7fe5980b4d60 2024-03-25T19:37:03.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.879+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073633111 0 0) 0x7fe5980af940 con 0x7fe59814be90 2024-03-25T19:37:03.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.879+0000 7fe595ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:03.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.879+0000 7fe595ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:03.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.879+0000 7fe595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:03.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.879+0000 7fe595ffb640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:37:03.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.879+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5980b0a70 con 0x7fe59814be90 2024-03-25T19:37:03.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.881+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654890732 0 0) 0x7fe5980b0d50 con 0x7fe5980b1730 2024-03-25T19:37:03.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.881+0000 7fe595ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:03.880 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.881+0000 7fe595ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:03.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.881+0000 7fe595ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:03.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.881+0000 7fe595ffb640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:37:03.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.881+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5980af940 con 0x7fe5980b1730 2024-03-25T19:37:03.886 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2618759343 0 0) 0x7fe580003520 con 0x7fe5980b4d60 2024-03-25T19:37:03.886 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 >> v1:172.21.15.143:6790/0 conn(0x7fe5980b1730 legacy=0x7fe5980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 >> v1:172.21.15.143:6789/0 conn(0x7fe59814be90 legacy=0x7fe59814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 1 monclient: found mon.noname-c 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5980b1030 con 0x7fe5980b4d60 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe588001690 con 0x7fe5980b4d60 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe595ffb640 10 monclient: dump: 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:03.887 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe59d2ef640 5 monclient: authenticate success, global_id 4543 2024-03-25T19:37:03.888 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.887+0000 7fe59d2ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.892+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5880017f0 con 0x7fe5980b4d60 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.892+0000 7fe595ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.892+0000 7fe595ffb640 1 -- 172.21.15.143:0/823145294 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe588001c00 con 0x7fe5980b4d60 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.892+0000 7fe595ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.892+0000 7fe595ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.892+0000 7fe595ffb640 10 monclient: dump: 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:03.892 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:03.893 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:03.893 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:03.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.892+0000 7fe59d2ef640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:03.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.892+0000 7fe59d2ef640 4 set_mon_vals no callback set 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.893+0000 7fe59d2ef640 10 monclient: shutdown 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.893+0000 7fe59d2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.893+0000 7fe59d2ef640 1 -- 172.21.15.143:0/823145294 >> v1:172.21.15.203:6789/0 conn(0x7fe5980b4d60 legacy=0x7fe59813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.893+0000 7fe59d2ef640 1 -- 172.21.15.143:0/823145294 shutdown_connections 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.893+0000 7fe59d2ef640 1 -- 172.21.15.143:0/823145294 >> 172.21.15.143:0/823145294 conn(0x7fe5980617a0 msgr2=0x7fe598061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.893+0000 7fe59d2ef640 1 -- 172.21.15.143:0/823145294 shutdown_connections 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.893+0000 7fe59d2ef640 1 -- 172.21.15.143:0/823145294 wait complete. 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.894+0000 7fe59d2ef640 10 monclient: build_initial_monmap 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.894+0000 7fe59d2ef640 10 monclient: monmap: 2024-03-25T19:37:03.894 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.894+0000 7fe59d2ef640 1 librados: starting msgr at 2024-03-25T19:37:03.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.894+0000 7fe59d2ef640 1 librados: starting objecter 2024-03-25T19:37:03.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 1 Processor -- start 2024-03-25T19:37:03.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 1 -- start start 2024-03-25T19:37:03.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 1 librados: setting wanted keys 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 1 librados: calling monclient init 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 10 monclient: init 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 10 monclient(hunting): picked mon.noname-c con 0x7fe5980b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 10 monclient(hunting): picked mon.noname-b con 0x7fe5980b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 10 monclient(hunting): picked mon.noname-a con 0x7fe59814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe598141f40 con 0x7fe59814be90 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe598142360 con 0x7fe5980b4d60 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5981424f0 con 0x7fe5980b1730 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:03.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.895+0000 7fe59d2ef640 10 monclient(hunting): authenticate will time out at 6440.239498s 2024-03-25T19:37:03.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.914+0000 7fe596ffd640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fe5980b1730 0x7fe5981405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:54748/0 (socket says 172.21.15.143:54748) 2024-03-25T19:37:03.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.914+0000 7fe596ffd640 1 -- 172.21.15.143:0/2857420343 learned_addr learned my addr 172.21.15.143:0/2857420343 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:03.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.915+0000 7fe57f7fe640 10 client.?.objecter ms_handle_connect 0x7fe5980b4d60 2024-03-25T19:37:03.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.915+0000 7fe57f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:03.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.916+0000 7fe57f7fe640 10 client.?.objecter ms_handle_connect 0x7fe5980b1730 2024-03-25T19:37:03.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.916+0000 7fe57f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:03.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.926+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276161508 0 0) 0x7fe5980b1030 con 0x7fe5980b1730 2024-03-25T19:37:03.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.926+0000 7fe57f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:03.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.926+0000 7fe57f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:03.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.926+0000 7fe57f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:03.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.926+0000 7fe57f7fe640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:37:03.926 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.926+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe584004eb0 con 0x7fe5980b1730 2024-03-25T19:37:03.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.929+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948357120 0 0) 0x7fe598142360 con 0x7fe5980b4d60 2024-03-25T19:37:03.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.929+0000 7fe57f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:03.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.929+0000 7fe57f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:03.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.929+0000 7fe57f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:03.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.929+0000 7fe57f7fe640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:37:03.929 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.929+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5980b1030 con 0x7fe5980b4d60 2024-03-25T19:37:03.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.932+0000 7fe57f7fe640 10 client.?.objecter ms_handle_connect 0x7fe59814be90 2024-03-25T19:37:03.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.932+0000 7fe57f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:03.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.944+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882108455 0 0) 0x7fe5980b1030 con 0x7fe5980b4d60 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.944+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe598142360 con 0x7fe5980b4d60 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.944+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe58c0018b0 con 0x7fe5980b4d60 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.944+0000 7fe57f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.944+0000 7fe57f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.944+0000 7fe57f7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:03.944 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:03.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.944+0000 7fe57f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:03.950 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916738079 0 0) 0x7fe598141f40 con 0x7fe59814be90 2024-03-25T19:37:03.950 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:03.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:03.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:03.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:37:03.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5980b1030 con 0x7fe59814be90 2024-03-25T19:37:03.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3923547047 0 0) 0x7fe598142360 con 0x7fe5980b4d60 2024-03-25T19:37:03.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 >> v1:172.21.15.143:6790/0 conn(0x7fe5980b1730 legacy=0x7fe5981405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:03.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 >> v1:172.21.15.143:6789/0 conn(0x7fe59814be90 legacy=0x7fe5981e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:03.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:03.952 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 1 monclient: found mon.b 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe598140d00 con 0x7fe5980b4d60 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe57f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 5 monclient: authenticate success, global_id 4546 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 10 monclient: _renew_subs 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe598138150 con 0x7fe5980b4d60 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 10 monclient: _renew_subs 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5981d8520 con 0x7fe5980b4d60 2024-03-25T19:37:03.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.951+0000 7fe59d2ef640 1 librados: init done 2024-03-25T19:37:03.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.952+0000 7fe59d2ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:03.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.953+0000 7fe59d2ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:03.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.953+0000 7fe59d2ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:37:03.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:03.953+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe598140f50 con 0x7fe5980b4d60 2024-03-25T19:37:04.227 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.228+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe58c003610 con 0x7fe5980b4d60 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.228+0000 7fe57f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.228+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fe58c0039a0 con 0x7fe5980b4d60 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.228+0000 7fe57f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.228+0000 7fe57f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.228+0000 7fe57f7fe640 10 monclient: dump: 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:04.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.228+0000 7fe5957fa640 4 set_mon_vals no callback set 2024-03-25T19:37:04.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.228+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fe58c003bc0 con 0x7fe5980b4d60 2024-03-25T19:37:04.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.229+0000 7fe596ffd640 10 monclient: get_auth_request con 0x7fe58406de10 auth_method 0 2024-03-25T19:37:04.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.229+0000 7fe57f7fe640 10 client.4546.objecter ms_handle_connect 0x7fe58406de10 2024-03-25T19:37:04.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.233+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fe58c0535c0 con 0x7fe5980b4d60 2024-03-25T19:37:04.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.233+0000 7fe57f7fe640 10 client.4546.objecter ms_dispatch 0x7fe598000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:04.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.233+0000 7fe57f7fe640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:04.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.233+0000 7fe57f7fe640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:04.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.233+0000 7fe57f7fe640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T19:37:04.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.234+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe58c04f0d0 con 0x7fe5980b4d60 2024-03-25T19:37:04.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.234+0000 7fe57f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:04.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.234+0000 7fe57f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:04.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.490+0000 7fe59d2ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.490+0000 7fe59d2ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.490+0000 7fe59d2ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:37:04.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.490+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5980627b0 con 0x7fe5980b4d60 2024-03-25T19:37:04.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.511+0000 7fe57f7fe640 1 -- 172.21.15.143:0/2857420343 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fe58c051000 con 0x7fe5980b4d60 2024-03-25T19:37:04.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.511+0000 7fe57f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:04.511 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.511+0000 7fe57f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:04.511 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:37:04.511 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.513+0000 7fe59d2ef640 10 librados: watch_flush enter 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.513+0000 7fe59d2ef640 10 librados: watch_flush exit 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.513+0000 7fe59d2ef640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.513+0000 7fe59d2ef640 10 client.4546.objecter successfully canceled tick 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.513+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 >> v1:172.21.15.143:6800/1292282666 conn(0x7fe58406de10 legacy=0x7fe5840702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.513+0000 7fe59d2ef640 10 monclient: shutdown 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.513+0000 7fe59d2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.513+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 >> v1:172.21.15.203:6789/0 conn(0x7fe5980b4d60 legacy=0x7fe5981d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.514+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 shutdown_connections 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.514+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 >> 172.21.15.143:0/2857420343 conn(0x7fe5980617a0 msgr2=0x7fe5980b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:04.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.514+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 shutdown_connections 2024-03-25T19:37:04.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.514+0000 7fe59d2ef640 1 -- 172.21.15.143:0/2857420343 wait complete. 2024-03-25T19:37:04.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:04.514+0000 7fe59d2ef640 1 librados: shutdown 2024-03-25T19:37:04.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:11.331 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:11.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.449+0000 7fc419264640 10 monclient: get_monmap_and_config 2024-03-25T19:37:11.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.449+0000 7fc419264640 10 monclient: build_initial_monmap 2024-03-25T19:37:11.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.449+0000 7fc419264640 10 monclient: monmap: 2024-03-25T19:37:11.449 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:11.449 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:37:11.450104+0000 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:37:11.450104+0000 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.449+0000 7fc419264640 1 Processor -- start 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.449+0000 7fc419264640 1 -- start start 2024-03-25T19:37:11.450 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.449+0000 7fc419264640 10 monclient: init 2024-03-25T19:37:11.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 10 monclient(hunting): picked mon.noname-b con 0x7fc4140a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 10 monclient(hunting): picked mon.noname-a con 0x7fc41413f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 10 monclient(hunting): picked mon.noname-c con 0x7fc4141430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc414142520 con 0x7fc41413f710 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4140b28a0 con 0x7fc4141430c0 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4140b2a30 con 0x7fc4140a98f0 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.450+0000 7fc419264640 10 monclient(hunting): authenticate will time out at 6447.794256s 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.452+0000 7fc412ffd640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fc4140a98f0 0x7fc4140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:58648/0 (socket says 172.21.15.143:58648) 2024-03-25T19:37:11.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.452+0000 7fc412ffd640 1 -- 172.21.15.143:0/2014112500 learned_addr learned my addr 172.21.15.143:0/2014112500 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:11.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.473+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028931300 0 0) 0x7fc4140b28a0 con 0x7fc4141430c0 2024-03-25T19:37:11.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.473+0000 7fc411ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:11.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.473+0000 7fc411ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:11.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.473+0000 7fc411ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:11.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.473+0000 7fc411ffb640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:37:11.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.473+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3fc003520 con 0x7fc4141430c0 2024-03-25T19:37:11.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.476+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524840403 0 0) 0x7fc414142520 con 0x7fc41413f710 2024-03-25T19:37:11.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.476+0000 7fc411ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:11.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.476+0000 7fc411ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:11.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.476+0000 7fc411ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:11.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.476+0000 7fc411ffb640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:37:11.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.476+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4140b28a0 con 0x7fc41413f710 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3123751132 0 0) 0x7fc4140b28a0 con 0x7fc41413f710 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 >> v1:172.21.15.143:6790/0 conn(0x7fc4140a98f0 legacy=0x7fc4140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 >> v1:172.21.15.203:6789/0 conn(0x7fc4141430c0 legacy=0x7fc41414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 1 monclient: found mon.noname-a 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4140b44c0 con 0x7fc41413f710 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc408001670 con 0x7fc41413f710 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:11.485 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.484+0000 7fc411ffb640 10 monclient: dump: 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.485+0000 7fc419264640 5 monclient: authenticate success, global_id 4548 2024-03-25T19:37:11.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.485+0000 7fc419264640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:11.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4080017d0 con 0x7fc41413f710 2024-03-25T19:37:11.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc411ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:11.488 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc419264640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc411ffb640 1 -- 172.21.15.143:0/2014112500 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc408001ba0 con 0x7fc41413f710 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc419264640 4 set_mon_vals no callback set 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc411ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc411ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc411ffb640 10 monclient: dump: 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:11.489 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:11.490 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:11.490 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:11.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc419264640 10 monclient: shutdown 2024-03-25T19:37:11.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc419264640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:11.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.489+0000 7fc419264640 1 -- 172.21.15.143:0/2014112500 >> v1:172.21.15.143:6789/0 conn(0x7fc41413f710 legacy=0x7fc4141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:11.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.490+0000 7fc419264640 1 -- 172.21.15.143:0/2014112500 shutdown_connections 2024-03-25T19:37:11.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.490+0000 7fc419264640 1 -- 172.21.15.143:0/2014112500 >> 172.21.15.143:0/2014112500 conn(0x7fc4140b7ea0 msgr2=0x7fc4140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:11.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.490+0000 7fc419264640 1 -- 172.21.15.143:0/2014112500 shutdown_connections 2024-03-25T19:37:11.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.490+0000 7fc419264640 1 -- 172.21.15.143:0/2014112500 wait complete. 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.490+0000 7fc419264640 10 monclient: build_initial_monmap 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.490+0000 7fc419264640 10 monclient: monmap: 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.491+0000 7fc419264640 1 librados: starting msgr at 2024-03-25T19:37:11.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.491+0000 7fc419264640 1 librados: starting objecter 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.491+0000 7fc419264640 1 Processor -- start 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.491+0000 7fc419264640 1 -- start start 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.491+0000 7fc419264640 1 librados: setting wanted keys 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.491+0000 7fc419264640 1 librados: calling monclient init 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.491+0000 7fc419264640 10 monclient: init 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 10 monclient(hunting): picked mon.noname-c con 0x7fc4140a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 10 monclient(hunting): picked mon.noname-a con 0x7fc41413f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 10 monclient(hunting): picked mon.noname-b con 0x7fc4141430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4141d8440 con 0x7fc41413f710 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4141d7810 con 0x7fc4141430c0 2024-03-25T19:37:11.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4141d50a0 con 0x7fc4140a98f0 2024-03-25T19:37:11.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:11.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc419264640 10 monclient(hunting): authenticate will time out at 6447.835808s 2024-03-25T19:37:11.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc412ffd640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fc4140a98f0 0x7fc4141d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:58692/0 (socket says 172.21.15.143:58692) 2024-03-25T19:37:11.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.492+0000 7fc412ffd640 1 -- 172.21.15.143:0/2502814956 learned_addr learned my addr 172.21.15.143:0/2502814956 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:11.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.493+0000 7fc3fb7fe640 10 client.?.objecter ms_handle_connect 0x7fc4140a98f0 2024-03-25T19:37:11.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.493+0000 7fc3fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:11.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.495+0000 7fc3fb7fe640 10 client.?.objecter ms_handle_connect 0x7fc4141430c0 2024-03-25T19:37:11.496 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.495+0000 7fc3fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:11.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.549+0000 7fc3fb7fe640 10 client.?.objecter ms_handle_connect 0x7fc41413f710 2024-03-25T19:37:11.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.549+0000 7fc3fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:11.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.573+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058575625 0 0) 0x7fc4141d8440 con 0x7fc41413f710 2024-03-25T19:37:11.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.573+0000 7fc3fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:11.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.573+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:11.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.573+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:11.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.573+0000 7fc3fb7fe640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:37:11.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.573+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc400004eb0 con 0x7fc41413f710 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.591+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 65875905 0 0) 0x7fc400004eb0 con 0x7fc41413f710 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.591+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4141d8440 con 0x7fc41413f710 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.591+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc408001ae0 con 0x7fc41413f710 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.591+0000 7fc3fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.591+0000 7fc3fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.591+0000 7fc3fb7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:11.591 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:11.592 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:11.592 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:11.592 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:11.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.591+0000 7fc3fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:11.602 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 779004684 0 0) 0x7fc4141d8440 con 0x7fc41413f710 2024-03-25T19:37:11.602 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 >> v1:172.21.15.143:6790/0 conn(0x7fc4140a98f0 legacy=0x7fc4141d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:11.602 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 >> v1:172.21.15.203:6789/0 conn(0x7fc4141430c0 legacy=0x7fc4141384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:11.602 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:11.602 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 1 monclient: found mon.a 2024-03-25T19:37:11.602 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:11.602 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4141d5230 con 0x7fc41413f710 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc3fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 5 monclient: authenticate success, global_id 4551 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 10 monclient: _renew_subs 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc414137f70 con 0x7fc41413f710 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 10 monclient: _renew_subs 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4141d41e0 con 0x7fc41413f710 2024-03-25T19:37:11.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.602+0000 7fc419264640 1 librados: init done 2024-03-25T19:37:11.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.603+0000 7fc419264640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:11.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.603+0000 7fc419264640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:11.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.603+0000 7fc419264640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:11.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.603+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4140abda0 con 0x7fc41413f710 2024-03-25T19:37:11.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.908+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4080038a0 con 0x7fc41413f710 2024-03-25T19:37:11.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.908+0000 7fc3fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:11.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.908+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fc408003bb0 con 0x7fc41413f710 2024-03-25T19:37:11.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.908+0000 7fc3fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.908+0000 7fc3fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.908+0000 7fc3fb7fe640 10 monclient: dump: 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:11.909 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:11.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.908+0000 7fc4117fa640 4 set_mon_vals no callback set 2024-03-25T19:37:11.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.908+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fc408019b80 con 0x7fc41413f710 2024-03-25T19:37:11.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.909+0000 7fc412ffd640 10 monclient: get_auth_request con 0x7fc40006dc70 auth_method 0 2024-03-25T19:37:11.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.909+0000 7fc3fb7fe640 10 client.4551.objecter ms_handle_connect 0x7fc40006dc70 2024-03-25T19:37:11.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.915+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fc408053710 con 0x7fc41413f710 2024-03-25T19:37:11.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.915+0000 7fc3fb7fe640 10 client.4551.objecter ms_dispatch 0x7fc414000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:11.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.915+0000 7fc3fb7fe640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:11.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.915+0000 7fc3fb7fe640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:11.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.915+0000 7fc3fb7fe640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T19:37:11.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.915+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc40804f220 con 0x7fc41413f710 2024-03-25T19:37:11.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.915+0000 7fc3fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:11.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:11.915+0000 7fc3fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:12.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.162+0000 7fc419264640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:12.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.162+0000 7fc419264640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:12.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.162+0000 7fc419264640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:12.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.162+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4141d79a0 con 0x7fc41413f710 2024-03-25T19:37:12.188 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.188+0000 7fc3fb7fe640 1 -- 172.21.15.143:0/2502814956 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fc408051150 con 0x7fc41413f710 2024-03-25T19:37:12.188 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.188+0000 7fc3fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:12.188 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.188+0000 7fc3fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:12.188 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:37:12.188 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.190+0000 7fc419264640 10 librados: watch_flush enter 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.190+0000 7fc419264640 10 librados: watch_flush exit 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.190+0000 7fc419264640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.190+0000 7fc419264640 10 client.4551.objecter successfully canceled tick 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 >> v1:172.21.15.143:6800/1292282666 conn(0x7fc40006dc70 legacy=0x7fc400070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 10 monclient: shutdown 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 >> v1:172.21.15.143:6789/0 conn(0x7fc41413f710 legacy=0x7fc4141d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 shutdown_connections 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 >> 172.21.15.143:0/2502814956 conn(0x7fc4140b7ea0 msgr2=0x7fc414074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:12.191 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 shutdown_connections 2024-03-25T19:37:12.192 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 1 -- 172.21.15.143:0/2502814956 wait complete. 2024-03-25T19:37:12.192 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:12.191+0000 7fc419264640 1 librados: shutdown 2024-03-25T19:37:12.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:19.009 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.125+0000 7f96531c8640 10 monclient: get_monmap_and_config 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.125+0000 7f96531c8640 10 monclient: build_initial_monmap 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.125+0000 7f96531c8640 10 monclient: monmap: 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:37:19.126488+0000 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:37:19.126488+0000 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:19.125 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:19.126 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:37:19.126 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:37:19.126 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.126+0000 7f96531c8640 1 Processor -- start 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.126+0000 7f96531c8640 1 -- start start 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.126+0000 7f96531c8640 10 monclient: init 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 10 monclient(hunting): picked mon.noname-a con 0x7f964c13fc60 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 10 monclient(hunting): picked mon.noname-c con 0x7f964c070570 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 10 monclient(hunting): picked mon.noname-b con 0x7f964c0727a0 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c13c9f0 con 0x7f964c13fc60 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c06fac0 con 0x7f964c070570 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c0715d0 con 0x7f964c0727a0 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:19.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.127+0000 7f96531c8640 10 monclient(hunting): authenticate will time out at 6455.470948s 2024-03-25T19:37:19.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.134+0000 7f964bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f964c070570 0x7f964c072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:39958/0 (socket says 172.21.15.143:39958) 2024-03-25T19:37:19.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.134+0000 7f964bfff640 1 -- 172.21.15.143:0/2363631082 learned_addr learned my addr 172.21.15.143:0/2363631082 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:19.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.153+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840631105 0 0) 0x7f964c13c9f0 con 0x7f964c13fc60 2024-03-25T19:37:19.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.153+0000 7f964b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:19.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.153+0000 7f964b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:19.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.153+0000 7f964b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:19.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.153+0000 7f964b7fe640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:37:19.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.153+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c003520 con 0x7f964c13fc60 2024-03-25T19:37:19.161 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:45.726 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4136344813 0 0) 0x7f963c003520 con 0x7f964c13fc60 2024-03-25T19:37:19.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 >> v1:172.21.15.143:6790/0 conn(0x7f964c0727a0 legacy=0x7f964c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:19.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 >> v1:172.21.15.203:6789/0 conn(0x7f964c070570 legacy=0x7f964c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:19.222 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:45.786 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:19.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:37:19.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:19.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:37:19.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c0718b0 con 0x7f964c13fc60 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f96531c8640 5 monclient: authenticate success, global_id 4554 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9640001690 con 0x7f964c13fc60 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f964b7fe640 10 monclient: dump: 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:19.223 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:19.224 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:19.224 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:19.224 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:19.224 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:19.224 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:19.224 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:19.224 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:19.224 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.222+0000 7f96531c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:19.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96400017f0 con 0x7f964c13fc60 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f964b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f964b7fe640 1 -- 172.21.15.143:0/2363631082 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9640001c00 con 0x7f964c13fc60 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f96531c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f96531c8640 4 set_mon_vals no callback set 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f964b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f964b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f964b7fe640 10 monclient: dump: 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:19.244 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:19.245 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:19.245 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:19.245 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:19.245 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.244+0000 7f96531c8640 10 monclient: shutdown 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.245+0000 7f96531c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.245+0000 7f96531c8640 1 -- 172.21.15.143:0/2363631082 >> v1:172.21.15.143:6789/0 conn(0x7f964c13fc60 legacy=0x7f964c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.245+0000 7f96531c8640 1 -- 172.21.15.143:0/2363631082 shutdown_connections 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.245+0000 7f96531c8640 1 -- 172.21.15.143:0/2363631082 >> 172.21.15.143:0/2363631082 conn(0x7f964c0b4a80 msgr2=0x7f964c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.245+0000 7f96531c8640 1 -- 172.21.15.143:0/2363631082 shutdown_connections 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.245+0000 7f96531c8640 1 -- 172.21.15.143:0/2363631082 wait complete. 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.245+0000 7f96531c8640 10 monclient: build_initial_monmap 2024-03-25T19:37:19.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.245+0000 7f96531c8640 10 monclient: monmap: 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.246+0000 7f96531c8640 1 librados: starting msgr at 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.246+0000 7f96531c8640 1 librados: starting objecter 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.246+0000 7f96531c8640 1 Processor -- start 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.246+0000 7f96531c8640 1 -- start start 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.246+0000 7f96531c8640 1 librados: setting wanted keys 2024-03-25T19:37:19.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.246+0000 7f96531c8640 1 librados: calling monclient init 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.246+0000 7f96531c8640 10 monclient: init 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 10 monclient(hunting): picked mon.noname-c con 0x7f964c070570 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 10 monclient(hunting): picked mon.noname-b con 0x7f964c0727a0 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 10 monclient(hunting): picked mon.noname-a con 0x7f964c13fc60 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c1345f0 con 0x7f964c13fc60 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c13b500 con 0x7f964c0727a0 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c13a060 con 0x7f964c070570 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:19.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.247+0000 7f96531c8640 10 monclient(hunting): authenticate will time out at 6455.590737s 2024-03-25T19:37:19.260 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:07:45.825 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.266+0000 7f9651765640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f964c13fc60 0x7f964c133ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:52150/0 (socket says 172.21.15.143:52150) 2024-03-25T19:37:19.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.266+0000 7f9651765640 1 -- 172.21.15.143:0/1570182214 learned_addr learned my addr 172.21.15.143:0/1570182214 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:19.270 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:45.806 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.280 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:45.815 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.285 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:45.850 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.287+0000 7f96497fa640 10 client.?.objecter ms_handle_connect 0x7f964c13fc60 2024-03-25T19:37:19.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.287+0000 7f96497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:19.290 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:45.855 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.299 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.299+0000 7f96497fa640 10 client.?.objecter ms_handle_connect 0x7f964c070570 2024-03-25T19:37:19.299 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.299+0000 7f96497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:19.299 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:07:45.835 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.307+0000 7f96497fa640 10 client.?.objecter ms_handle_connect 0x7f964c0727a0 2024-03-25T19:37:19.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.307+0000 7f96497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:19.315 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:07:45.880 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:19.316 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.316+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77167236 0 0) 0x7f964c13a060 con 0x7f964c070570 2024-03-25T19:37:19.316 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.316+0000 7f96497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:19.316 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.316+0000 7f96497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:19.316 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.316+0000 7f96497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:19.316 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.316+0000 7f96497fa640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:37:19.316 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.316+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9630004eb0 con 0x7f964c070570 2024-03-25T19:37:19.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.335+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959908998 0 0) 0x7f964c1345f0 con 0x7f964c13fc60 2024-03-25T19:37:19.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.335+0000 7f96497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:19.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.335+0000 7f96497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:19.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.335+0000 7f96497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:19.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.335+0000 7f96497fa640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:37:19.335 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.335+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f964c13a060 con 0x7f964c13fc60 2024-03-25T19:37:19.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.343+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270344061 0 0) 0x7f964c13b500 con 0x7f964c0727a0 2024-03-25T19:37:19.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.343+0000 7f96497fa640 10 monclient(hunting): _negotiate 2024-03-25T19:37:19.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.343+0000 7f96497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:19.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.343+0000 7f96497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:19.343 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.343+0000 7f96497fa640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:37:19.343 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.343+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f964c1345f0 con 0x7f964c0727a0 2024-03-25T19:37:19.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.357+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2768697998 0 0) 0x7f9630004eb0 con 0x7f964c070570 2024-03-25T19:37:19.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.357+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f964c13b500 con 0x7f964c070570 2024-03-25T19:37:19.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.357+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9640001bc0 con 0x7f964c070570 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.357+0000 7f96497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.357+0000 7f96497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.357+0000 7f96497fa640 10 monclient(hunting): dump: 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:19.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.358+0000 7f96497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.370+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 919890862 0 0) 0x7f964c13b500 con 0x7f964c070570 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 >> v1:172.21.15.203:6789/0 conn(0x7f964c0727a0 legacy=0x7f964c1337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 >> v1:172.21.15.143:6789/0 conn(0x7f964c13fc60 legacy=0x7f964c133ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 10 monclient: _finish_hunting 0 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 1 monclient: found mon.c 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c13a1f0 con 0x7f964c070570 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 10 monclient: _finish_auth 0 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 10 monclient: _check_auth_tickets 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 5 monclient: authenticate success, global_id 4544 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 10 monclient: _renew_subs 2024-03-25T19:37:19.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:19.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c13bb90 con 0x7f964c070570 2024-03-25T19:37:19.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:19.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 10 monclient: _renew_subs 2024-03-25T19:37:19.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:19.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c13b1c0 con 0x7f964c070570 2024-03-25T19:37:19.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.371+0000 7f96531c8640 1 librados: init done 2024-03-25T19:37:19.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.372+0000 7f96531c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:19.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.372+0000 7f96531c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:19.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.372+0000 7f96531c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:19.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.372+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c0627b0 con 0x7f964c070570 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.676+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96400038a0 con 0x7f964c070570 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.676+0000 7f96497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.676+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9640003bb0 con 0x7f964c070570 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.676+0000 7f96497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.676+0000 7f96497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.676+0000 7f96497fa640 10 monclient: dump: 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:19.676 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.676+0000 7f964affd640 4 set_mon_vals no callback set 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.676+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f9640019b80 con 0x7f964c070570 2024-03-25T19:37:19.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.677+0000 7f964bfff640 10 monclient: get_auth_request con 0x7f963006ddf0 auth_method 0 2024-03-25T19:37:19.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.677+0000 7f96497fa640 10 client.4544.objecter ms_handle_connect 0x7f963006ddf0 2024-03-25T19:37:19.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.691+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f9640053740 con 0x7f964c070570 2024-03-25T19:37:19.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.691+0000 7f96497fa640 10 client.4544.objecter ms_dispatch 0x7f964c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:19.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.691+0000 7f96497fa640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:19.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.691+0000 7f96497fa640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:19.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.691+0000 7f96497fa640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-25T19:37:19.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.691+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f964004f080 con 0x7f964c070570 2024-03-25T19:37:19.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.691+0000 7f96497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:19.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.691+0000 7f96497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:19.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.936+0000 7f96531c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:19.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.936+0000 7f96531c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:19.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.936+0000 7f96531c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:19.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.936+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f964c13b690 con 0x7f964c070570 2024-03-25T19:37:19.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.958+0000 7f96497fa640 1 -- 172.21.15.143:0/1570182214 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f9640051180 con 0x7f964c070570 2024-03-25T19:37:19.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.958+0000 7f96497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:19.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.958+0000 7f96497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:19.958 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:37:19.959 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:19.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 10 librados: watch_flush enter 2024-03-25T19:37:19.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 10 librados: watch_flush exit 2024-03-25T19:37:19.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-25T19:37:19.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 10 client.4544.objecter successfully canceled tick 2024-03-25T19:37:19.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 >> v1:172.21.15.143:6800/1292282666 conn(0x7f963006ddf0 legacy=0x7f96300702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:19.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 10 monclient: shutdown 2024-03-25T19:37:19.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:19.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 >> v1:172.21.15.143:6790/0 conn(0x7f964c070570 legacy=0x7f964c1d5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:19.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 shutdown_connections 2024-03-25T19:37:19.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 >> 172.21.15.143:0/1570182214 conn(0x7f964c0b4a80 msgr2=0x7f964c0b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:19.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 shutdown_connections 2024-03-25T19:37:19.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 1 -- 172.21.15.143:0/1570182214 wait complete. 2024-03-25T19:37:19.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:19.961+0000 7f96531c8640 1 librados: shutdown 2024-03-25T19:37:19.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:26.780 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.897+0000 7f766d401640 10 monclient: get_monmap_and_config 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.897+0000 7f766d401640 10 monclient: build_initial_monmap 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.897+0000 7f766d401640 10 monclient: monmap: 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:37:26.899331+0000 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:37:26.899331+0000 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:37:26.898 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:37:26.899 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.898+0000 7f766d401640 1 Processor -- start 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.898+0000 7f766d401640 1 -- start start 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.898+0000 7f766d401640 10 monclient: init 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 10 monclient(hunting): picked mon.noname-c con 0x7f76680b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 10 monclient(hunting): picked mon.noname-a con 0x7f76680b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 10 monclient(hunting): picked mon.noname-b con 0x7f766814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76680af940 con 0x7f76680b1730 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76680b0a70 con 0x7f76680b4d60 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76680b0d50 con 0x7f766814be90 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:26.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.899+0000 7f766d401640 10 monclient(hunting): authenticate will time out at 6463.244016s 2024-03-25T19:37:26.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.909+0000 7f76677fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f766814be90 0x7f766814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:50732/0 (socket says 172.21.15.143:50732) 2024-03-25T19:37:26.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.909+0000 7f76677fe640 1 -- 172.21.15.143:0/2526142195 learned_addr learned my addr 172.21.15.143:0/2526142195 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:26.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.914+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467671688 0 0) 0x7f76680af940 con 0x7f76680b1730 2024-03-25T19:37:26.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.914+0000 7f7665ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:26.914 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.914+0000 7f7665ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:26.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.914+0000 7f7665ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:26.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.914+0000 7f7665ffb640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:37:26.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.914+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7650003520 con 0x7f76680b1730 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222358265 0 0) 0x7f7650003520 con 0x7f76680b1730 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 >> v1:172.21.15.143:6790/0 conn(0x7f766814be90 legacy=0x7f766814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 >> v1:172.21.15.203:6789/0 conn(0x7f76680b4d60 legacy=0x7f766813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 1 monclient: found mon.noname-a 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680b1030 con 0x7f76680b1730 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:26.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f765c001690 con 0x7f76680b1730 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f7665ffb640 10 monclient: dump: 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f766d401640 5 monclient: authenticate success, global_id 4560 2024-03-25T19:37:26.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.919+0000 7f766d401640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:26.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.930+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f765c0017f0 con 0x7f76680b1730 2024-03-25T19:37:26.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.930+0000 7f7665ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:26.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.930+0000 7f7665ffb640 1 -- 172.21.15.143:0/2526142195 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f765c001c00 con 0x7f76680b1730 2024-03-25T19:37:26.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.930+0000 7f7665ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:26.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.930+0000 7f7665ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:26.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.930+0000 7f7665ffb640 10 monclient: dump: 2024-03-25T19:37:26.931 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.930+0000 7f766d401640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:26.932 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.930+0000 7f766d401640 4 set_mon_vals no callback set 2024-03-25T19:37:26.933 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.931+0000 7f766d401640 10 monclient: shutdown 2024-03-25T19:37:26.933 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.931+0000 7f766d401640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:26.933 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.931+0000 7f766d401640 1 -- 172.21.15.143:0/2526142195 >> v1:172.21.15.143:6789/0 conn(0x7f76680b1730 legacy=0x7f76680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.931+0000 7f766d401640 1 -- 172.21.15.143:0/2526142195 shutdown_connections 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.931+0000 7f766d401640 1 -- 172.21.15.143:0/2526142195 >> 172.21.15.143:0/2526142195 conn(0x7f76680617a0 msgr2=0x7f7668061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.931+0000 7f766d401640 1 -- 172.21.15.143:0/2526142195 shutdown_connections 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.931+0000 7f766d401640 1 -- 172.21.15.143:0/2526142195 wait complete. 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.932+0000 7f766d401640 10 monclient: build_initial_monmap 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.932+0000 7f766d401640 10 monclient: monmap: 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:37:26.934 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.932+0000 7f766d401640 1 librados: starting msgr at 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.932+0000 7f766d401640 1 librados: starting objecter 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 1 Processor -- start 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 1 -- start start 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 1 librados: setting wanted keys 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 1 librados: calling monclient init 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 10 monclient: init 2024-03-25T19:37:26.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 10 monclient(hunting): picked mon.noname-a con 0x7f76680b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 10 monclient(hunting): picked mon.noname-b con 0x7f76680b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 10 monclient(hunting): picked mon.noname-c con 0x7f766814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.933+0000 7f766d401640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76681d7940 con 0x7f76680b1730 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.934+0000 7f766d401640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76681d8570 con 0x7f76680b4d60 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.934+0000 7f766d401640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76681d8700 con 0x7f766814be90 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.934+0000 7f766d401640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:26.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.934+0000 7f766d401640 10 monclient(hunting): authenticate will time out at 6463.278702s 2024-03-25T19:37:26.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.944+0000 7f76667fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f76680b4d60 0x7f7668136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:52822/0 (socket says 172.21.15.143:52822) 2024-03-25T19:37:26.945 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.944+0000 7f76667fc640 1 -- 172.21.15.143:0/1400659304 learned_addr learned my addr 172.21.15.143:0/1400659304 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:26.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.948+0000 7f764f7fe640 10 client.?.objecter ms_handle_connect 0x7f76680b4d60 2024-03-25T19:37:26.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.948+0000 7f764f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:26.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.950+0000 7f764f7fe640 10 client.?.objecter ms_handle_connect 0x7f76680b1730 2024-03-25T19:37:26.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.950+0000 7f764f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:26.955 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.955+0000 7f764f7fe640 10 client.?.objecter ms_handle_connect 0x7f766814be90 2024-03-25T19:37:26.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.955+0000 7f764f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:26.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680009626 0 0) 0x7f76681d8700 con 0x7f766814be90 2024-03-25T19:37:26.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:26.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:26.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:26.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:37:26.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7654004eb0 con 0x7f766814be90 2024-03-25T19:37:26.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964395422 0 0) 0x7f76681d7940 con 0x7f76680b1730 2024-03-25T19:37:26.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:26.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:26.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:26.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:37:26.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.968+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76681d8700 con 0x7f76680b1730 2024-03-25T19:37:26.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1586532090 0 0) 0x7f7654004eb0 con 0x7f766814be90 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76681d7940 con 0x7f766814be90 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486010215 0 0) 0x7f76681d8570 con 0x7f76680b4d60 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7654004eb0 con 0x7f76680b4d60 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7660001b00 con 0x7f766814be90 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 10 monclient(hunting): dump: 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:26.978 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:26.979 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:26.979 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:26.979 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:26.979 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:26.979 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:26.979 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:26.979 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:26.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.977+0000 7f764f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:26.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2217564667 0 0) 0x7f76681d7940 con 0x7f766814be90 2024-03-25T19:37:26.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 >> v1:172.21.15.203:6789/0 conn(0x7f76680b4d60 legacy=0x7f7668136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:26.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 >> v1:172.21.15.143:6789/0 conn(0x7f76680b1730 legacy=0x7f76681d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:26.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:37:26.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 1 monclient: found mon.c 2024-03-25T19:37:26.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76681361c0 con 0x7f766814be90 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f764f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 5 monclient: authenticate success, global_id 4550 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 10 monclient: _renew_subs 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7668136470 con 0x7f766814be90 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 10 monclient: _renew_subs 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7668137790 con 0x7f766814be90 2024-03-25T19:37:26.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.989+0000 7f766d401640 1 librados: init done 2024-03-25T19:37:26.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.991+0000 7f766d401640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:26.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.991+0000 7f766d401640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:26.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.991+0000 7f766d401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:26.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:26.991+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76680627b0 con 0x7f766814be90 2024-03-25T19:37:27.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.291+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7660002030 con 0x7f766814be90 2024-03-25T19:37:27.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.291+0000 7f764f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:27.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.291+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7660004d00 con 0x7f766814be90 2024-03-25T19:37:27.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.291+0000 7f764f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:27.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.291+0000 7f764f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.291+0000 7f764f7fe640 10 monclient: dump: 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:27.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.291+0000 7f76657fa640 4 set_mon_vals no callback set 2024-03-25T19:37:27.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.292+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f7660004f20 con 0x7f766814be90 2024-03-25T19:37:27.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.293+0000 7f7666ffd640 10 monclient: get_auth_request con 0x7f765406de50 auth_method 0 2024-03-25T19:37:27.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.293+0000 7f764f7fe640 10 client.4550.objecter ms_handle_connect 0x7f765406de50 2024-03-25T19:37:27.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.300+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f7660054980 con 0x7f766814be90 2024-03-25T19:37:27.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.300+0000 7f764f7fe640 10 client.4550.objecter ms_dispatch 0x7f7668000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:27.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.300+0000 7f764f7fe640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:27.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.300+0000 7f764f7fe640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:27.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.300+0000 7f764f7fe640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T19:37:27.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.300+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76600502c0 con 0x7f766814be90 2024-03-25T19:37:27.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.300+0000 7f764f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:27.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.300+0000 7f764f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:27.551 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.550+0000 7f766d401640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:27.551 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.550+0000 7f766d401640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:27.551 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.550+0000 7f766d401640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:37:27.551 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.550+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76681d77a0 con 0x7f766814be90 2024-03-25T19:37:27.590 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.589+0000 7f764f7fe640 1 -- 172.21.15.143:0/1400659304 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f76681d77a0 con 0x7f766814be90 2024-03-25T19:37:27.590 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.589+0000 7f764f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:27.590 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.589+0000 7f764f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:27.590 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:37:27.591 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:27.593 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 10 librados: watch_flush enter 2024-03-25T19:37:27.593 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 10 librados: watch_flush exit 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 10 client.4550.objecter successfully canceled tick 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 >> v1:172.21.15.143:6800/1292282666 conn(0x7f765406de50 legacy=0x7f7654070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 10 monclient: shutdown 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 >> v1:172.21.15.143:6790/0 conn(0x7f766814be90 legacy=0x7f766812fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 shutdown_connections 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 >> 172.21.15.143:0/1400659304 conn(0x7f76680617a0 msgr2=0x7f7668061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 shutdown_connections 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 1 -- 172.21.15.143:0/1400659304 wait complete. 2024-03-25T19:37:27.594 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:27.592+0000 7f766d401640 1 librados: shutdown 2024-03-25T19:37:27.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:34.412 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:34.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.526+0000 7f031f538640 10 monclient: get_monmap_and_config 2024-03-25T19:37:34.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.526+0000 7f031f538640 10 monclient: build_initial_monmap 2024-03-25T19:37:34.526 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.526+0000 7f031f538640 10 monclient: monmap: 2024-03-25T19:37:34.526 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:34.526 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:34.527 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:37:34.527798+0000 2024-03-25T19:37:34.527 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:37:34.527798+0000 2024-03-25T19:37:34.527 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:34.527 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:34.527 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:34.527 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:37:34.527 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:37:34.527 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:34.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.527+0000 7f031f538640 1 Processor -- start 2024-03-25T19:37:34.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.527+0000 7f031f538640 1 -- start start 2024-03-25T19:37:34.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.527+0000 7f031f538640 10 monclient: init 2024-03-25T19:37:34.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:34.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:34.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 10 monclient(hunting): picked mon.noname-b con 0x7f03180a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:34.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 10 monclient(hunting): picked mon.noname-c con 0x7f031813f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:34.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 10 monclient(hunting): picked mon.noname-a con 0x7f03181430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:34.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0318142520 con 0x7f03181430c0 2024-03-25T19:37:34.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03180b28a0 con 0x7f031813f710 2024-03-25T19:37:34.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03180b2a30 con 0x7f03180a98f0 2024-03-25T19:37:34.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:34.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.528+0000 7f031f538640 10 monclient(hunting): authenticate will time out at 6470.871962s 2024-03-25T19:37:34.538 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.539+0000 7f031dad5640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f03181430c0 0x7f031814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:43808/0 (socket says 172.21.15.143:43808) 2024-03-25T19:37:34.539 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.539+0000 7f031dad5640 1 -- 172.21.15.143:0/3557778132 learned_addr learned my addr 172.21.15.143:0/3557778132 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:34.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.543+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769826487 0 0) 0x7f0318142520 con 0x7f03181430c0 2024-03-25T19:37:34.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.543+0000 7f030ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:34.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.543+0000 7f030ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:34.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.543+0000 7f030ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:34.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.543+0000 7f030ffff640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:37:34.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.543+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0304003520 con 0x7f03181430c0 2024-03-25T19:37:34.551 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.552+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599455743 0 0) 0x7f03180b2a30 con 0x7f03180a98f0 2024-03-25T19:37:34.552 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.552+0000 7f030ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:34.552 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.552+0000 7f030ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:34.552 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.552+0000 7f030ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:34.552 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.552+0000 7f030ffff640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:37:34.552 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.552+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0318142520 con 0x7f03180a98f0 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236393024 0 0) 0x7f0304003520 con 0x7f03181430c0 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 >> v1:172.21.15.143:6790/0 conn(0x7f03180a98f0 legacy=0x7f03180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 >> v1:172.21.15.203:6789/0 conn(0x7f031813f710 legacy=0x7f03181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 1 monclient: found mon.noname-a 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180b44c0 con 0x7f03181430c0 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 10 monclient: _finish_auth 0 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f030ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:34.553 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f031f538640 5 monclient: authenticate success, global_id 4566 2024-03-25T19:37:34.554 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.553+0000 7f031f538640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:34.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f03100019d0 con 0x7f03181430c0 2024-03-25T19:37:34.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:34.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:34.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 10 monclient: dump: 2024-03-25T19:37:34.557 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:34.557 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0310001c20 con 0x7f03181430c0 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f031f538640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f031f538640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 1 -- 172.21.15.143:0/3557778132 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0310001fb0 con 0x7f03181430c0 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f031f538640 4 set_mon_vals no callback set 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:34.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.557+0000 7f030ffff640 10 monclient: dump: 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:34.559 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.558+0000 7f031f538640 10 monclient: shutdown 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.558+0000 7f031f538640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.558+0000 7f031f538640 1 -- 172.21.15.143:0/3557778132 >> v1:172.21.15.143:6789/0 conn(0x7f03181430c0 legacy=0x7f031814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.558+0000 7f031f538640 1 -- 172.21.15.143:0/3557778132 shutdown_connections 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.558+0000 7f031f538640 1 -- 172.21.15.143:0/3557778132 >> 172.21.15.143:0/3557778132 conn(0x7f03180b7ea0 msgr2=0x7f03180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.558+0000 7f031f538640 1 -- 172.21.15.143:0/3557778132 shutdown_connections 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.558+0000 7f031f538640 1 -- 172.21.15.143:0/3557778132 wait complete. 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.559+0000 7f031f538640 10 monclient: build_initial_monmap 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.559+0000 7f031f538640 10 monclient: monmap: 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:34.561 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 1 librados: starting msgr at 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 1 librados: starting objecter 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 1 Processor -- start 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 1 -- start start 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 1 librados: setting wanted keys 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 1 librados: calling monclient init 2024-03-25T19:37:34.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 10 monclient: init 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 10 monclient(hunting): picked mon.noname-b con 0x7f03180a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 10 monclient(hunting): picked mon.noname-a con 0x7f031813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 10 monclient(hunting): picked mon.noname-c con 0x7f03181430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.560+0000 7f031f538640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03181378c0 con 0x7f031813f710 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.561+0000 7f031f538640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03181dc070 con 0x7f03180a98f0 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.561+0000 7f031f538640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0318138ea0 con 0x7f03181430c0 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.561+0000 7f031f538640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:34.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.561+0000 7f031f538640 10 monclient(hunting): authenticate will time out at 6470.904683s 2024-03-25T19:37:34.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.569+0000 7f031d2d4640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f03180a98f0 0x7f0318137d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:57678/0 (socket says 172.21.15.143:57678) 2024-03-25T19:37:34.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.569+0000 7f031d2d4640 1 -- 172.21.15.143:0/894738138 learned_addr learned my addr 172.21.15.143:0/894738138 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:34.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.569+0000 7f030dffb640 10 client.?.objecter ms_handle_connect 0x7f03180a98f0 2024-03-25T19:37:34.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.569+0000 7f030dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:34.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.570+0000 7f030dffb640 10 client.?.objecter ms_handle_connect 0x7f03181430c0 2024-03-25T19:37:34.570 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.571+0000 7f030dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:34.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.578+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406366933 0 0) 0x7f03181dc070 con 0x7f03180a98f0 2024-03-25T19:37:34.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.578+0000 7f030dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:34.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.578+0000 7f030dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:34.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.578+0000 7f030dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:34.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.578+0000 7f030dffb640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:37:34.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.578+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02fc004eb0 con 0x7f03180a98f0 2024-03-25T19:37:34.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.579+0000 7f030dffb640 10 client.?.objecter ms_handle_connect 0x7f031813f710 2024-03-25T19:37:34.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.579+0000 7f030dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:34.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.591+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647330835 0 0) 0x7f0318138ea0 con 0x7f03181430c0 2024-03-25T19:37:34.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.591+0000 7f030dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:34.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.591+0000 7f030dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:34.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.591+0000 7f030dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:34.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.591+0000 7f030dffb640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:37:34.591 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.591+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03181dc070 con 0x7f03181430c0 2024-03-25T19:37:34.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.596+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072472000 0 0) 0x7f03181378c0 con 0x7f031813f710 2024-03-25T19:37:34.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.596+0000 7f030dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:34.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.596+0000 7f030dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:34.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.596+0000 7f030dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:34.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.596+0000 7f030dffb640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:37:34.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.596+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0318138ea0 con 0x7f031813f710 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.603+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1563666156 0 0) 0x7f02fc004eb0 con 0x7f03180a98f0 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.603+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03181378c0 con 0x7f03180a98f0 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.603+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0314001a10 con 0x7f03180a98f0 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.603+0000 7f030dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.603+0000 7f030dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.603+0000 7f030dffb640 10 monclient(hunting): dump: 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:34.603 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.604 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.604 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:34.604 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:34.604 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:34.604 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:34.604 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:34.604 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:34.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.603+0000 7f030dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:34.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.609+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047732824 0 0) 0x7f03181dc070 con 0x7f03181430c0 2024-03-25T19:37:34.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.609+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02fc004eb0 con 0x7f03181430c0 2024-03-25T19:37:34.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.609+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0310001f30 con 0x7f03181430c0 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.609+0000 7f030dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.609+0000 7f030dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.609+0000 7f030dffb640 10 monclient(hunting): dump: 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.610+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3982920156 0 0) 0x7f0318138ea0 con 0x7f031813f710 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.610+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03181dc070 con 0x7f031813f710 2024-03-25T19:37:34.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.610+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0308001860 con 0x7f031813f710 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.610+0000 7f030dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.610+0000 7f030dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.610+0000 7f030dffb640 10 monclient(hunting): dump: 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:34.611 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:34.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3659446268 0 0) 0x7f03181dc070 con 0x7f031813f710 2024-03-25T19:37:34.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 >> v1:172.21.15.143:6790/0 conn(0x7f03181430c0 legacy=0x7f03181db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:34.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 >> v1:172.21.15.203:6789/0 conn(0x7f03180a98f0 legacy=0x7f0318137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:34.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 1 monclient: found mon.a 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0318139180 con 0x7f031813f710 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f030dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 5 monclient: authenticate success, global_id 4569 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 10 monclient: _renew_subs 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03181398f0 con 0x7f031813f710 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:34.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 10 monclient: _renew_subs 2024-03-25T19:37:34.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:34.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03181374a0 con 0x7f031813f710 2024-03-25T19:37:34.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.622+0000 7f031f538640 1 librados: init done 2024-03-25T19:37:34.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.623+0000 7f031f538640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:34.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.623+0000 7f031f538640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:34.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.623+0000 7f031f538640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:34.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.623+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03180627b0 con 0x7f031813f710 2024-03-25T19:37:34.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.871+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03080035f0 con 0x7f031813f710 2024-03-25T19:37:34.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.871+0000 7f030dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:34.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.871+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0308003900 con 0x7f031813f710 2024-03-25T19:37:34.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.871+0000 7f030dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:34.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.871+0000 7f030dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:34.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.871+0000 7f030dffb640 10 monclient: dump: 2024-03-25T19:37:34.871 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:34.871 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:34.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.871+0000 7f030f7fe640 4 set_mon_vals no callback set 2024-03-25T19:37:34.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.871+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f03080198d0 con 0x7f031813f710 2024-03-25T19:37:34.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.872+0000 7f031d2d4640 10 monclient: get_auth_request con 0x7f02fc06e160 auth_method 0 2024-03-25T19:37:34.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.872+0000 7f030dffb640 10 client.4569.objecter ms_handle_connect 0x7f02fc06e160 2024-03-25T19:37:34.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.877+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f0308053500 con 0x7f031813f710 2024-03-25T19:37:34.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.877+0000 7f030dffb640 10 client.4569.objecter ms_dispatch 0x7f0318000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:34.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.877+0000 7f030dffb640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:34.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.877+0000 7f030dffb640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:34.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.877+0000 7f030dffb640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-25T19:37:34.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.877+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f030804f010 con 0x7f031813f710 2024-03-25T19:37:34.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.877+0000 7f030dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:34.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:34.877+0000 7f030dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:35.105 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.105+0000 7f031f538640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:35.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.106+0000 7f031f538640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:35.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.106+0000 7f031f538640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:35.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.106+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f031814e980 con 0x7f031813f710 2024-03-25T19:37:35.130 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.130+0000 7f030dffb640 1 -- 172.21.15.143:0/894738138 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f0308050f40 con 0x7f031813f710 2024-03-25T19:37:35.130 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.130+0000 7f030dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:35.130 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.130+0000 7f030dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:35.130 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:37:35.130 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.132+0000 7f031f538640 10 librados: watch_flush enter 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.132+0000 7f031f538640 10 librados: watch_flush exit 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.132+0000 7f031f538640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.132+0000 7f031f538640 10 client.4569.objecter successfully canceled tick 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 >> v1:172.21.15.143:6800/1292282666 conn(0x7f02fc06e160 legacy=0x7f02fc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 10 monclient: shutdown 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 >> v1:172.21.15.143:6789/0 conn(0x7f031813f710 legacy=0x7f03181db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 shutdown_connections 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 >> 172.21.15.143:0/894738138 conn(0x7f03180b7ea0 msgr2=0x7f0318074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:35.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 shutdown_connections 2024-03-25T19:37:35.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 1 -- 172.21.15.143:0/894738138 wait complete. 2024-03-25T19:37:35.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:35.133+0000 7f031f538640 1 librados: shutdown 2024-03-25T19:37:35.145 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:39.250 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:05.814 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:39.260 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:05.825 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:39.290 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:05.855 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:39.308 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:05.844 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:39.314 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:05.879 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:39.316 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:05.852 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:39.330 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:05.895 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:39.350 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:05.886 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:39.366 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:05.930 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:41.952 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.069+0000 7f1e0c515640 10 monclient: get_monmap_and_config 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.069+0000 7f1e0c515640 10 monclient: build_initial_monmap 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.069+0000 7f1e0c515640 10 monclient: monmap: 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:37:42.070683+0000 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:37:42.070683+0000 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:42.069 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:42.070 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:42.070 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:37:42.070 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:37:42.070 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:42.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.070+0000 7f1e0c515640 1 Processor -- start 2024-03-25T19:37:42.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.070+0000 7f1e0c515640 1 -- start start 2024-03-25T19:37:42.070 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.070+0000 7f1e0c515640 10 monclient: init 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 10 monclient(hunting): picked mon.noname-a con 0x7f1e040a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 10 monclient(hunting): picked mon.noname-b con 0x7f1e0413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 10 monclient(hunting): picked mon.noname-c con 0x7f1e041430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e04142520 con 0x7f1e040a98f0 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e040b28a0 con 0x7f1e041430c0 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e040b2a30 con 0x7f1e0413f710 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:42.071 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.071+0000 7f1e0c515640 10 monclient(hunting): authenticate will time out at 6478.415591s 2024-03-25T19:37:42.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.080+0000 7f1e09ab0640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f1e0413f710 0x7f1e041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:43872/0 (socket says 172.21.15.143:43872) 2024-03-25T19:37:42.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.080+0000 7f1e09ab0640 1 -- 172.21.15.143:0/4242836761 learned_addr learned my addr 172.21.15.143:0/4242836761 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:42.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.093+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335459913 0 0) 0x7f1e040b2a30 con 0x7f1e0413f710 2024-03-25T19:37:42.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.093+0000 7f1e092af640 10 monclient(hunting): _negotiate 2024-03-25T19:37:42.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.093+0000 7f1e092af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:42.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.093+0000 7f1e092af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:42.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.093+0000 7f1e092af640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:37:42.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.093+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec003520 con 0x7f1e0413f710 2024-03-25T19:37:42.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.100+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172698903 0 0) 0x7f1e040b28a0 con 0x7f1e041430c0 2024-03-25T19:37:42.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.100+0000 7f1e092af640 10 monclient(hunting): _negotiate 2024-03-25T19:37:42.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.100+0000 7f1e092af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:42.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.100+0000 7f1e092af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:42.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.100+0000 7f1e092af640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:37:42.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.100+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e040b2a30 con 0x7f1e041430c0 2024-03-25T19:37:42.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 504391324 0 0) 0x7f1dec003520 con 0x7f1e0413f710 2024-03-25T19:37:42.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 >> v1:172.21.15.203:6789/0 conn(0x7f1e041430c0 legacy=0x7f1e0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:42.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 >> v1:172.21.15.143:6789/0 conn(0x7f1e040a98f0 legacy=0x7f1e040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:42.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 10 monclient: _finish_hunting 0 2024-03-25T19:37:42.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 1 monclient: found mon.noname-b 2024-03-25T19:37:42.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:42.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:37:42.106 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e040b44c0 con 0x7f1e0413f710 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 10 monclient: _finish_auth 0 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 10 monclient: _check_auth_tickets 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1df8001690 con 0x7f1e0413f710 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e092af640 10 monclient: dump: 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:42.107 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:42.108 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:42.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e0c515640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:37:42.108 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.106+0000 7f1e0c515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:42.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.117+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1df80017f0 con 0x7f1e0413f710 2024-03-25T19:37:42.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.117+0000 7f1e092af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.117+0000 7f1e092af640 1 -- 172.21.15.143:0/4242836761 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1df8001c00 con 0x7f1e0413f710 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.117+0000 7f1e0c515640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.117+0000 7f1e0c515640 4 set_mon_vals no callback set 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.117+0000 7f1e092af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.117+0000 7f1e092af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.118+0000 7f1e092af640 10 monclient: dump: 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:42.118 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:42.119 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:42.119 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:42.119 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.118+0000 7f1e0c515640 10 monclient: shutdown 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.118+0000 7f1e0c515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.118+0000 7f1e0c515640 1 -- 172.21.15.143:0/4242836761 >> v1:172.21.15.143:6790/0 conn(0x7f1e0413f710 legacy=0x7f1e041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.118+0000 7f1e0c515640 1 -- 172.21.15.143:0/4242836761 shutdown_connections 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.118+0000 7f1e0c515640 1 -- 172.21.15.143:0/4242836761 >> 172.21.15.143:0/4242836761 conn(0x7f1e040b7ea0 msgr2=0x7f1e040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.119+0000 7f1e0c515640 1 -- 172.21.15.143:0/4242836761 shutdown_connections 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.119+0000 7f1e0c515640 1 -- 172.21.15.143:0/4242836761 wait complete. 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.119+0000 7f1e0c515640 10 monclient: build_initial_monmap 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.119+0000 7f1e0c515640 10 monclient: monmap: 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:42.122 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 1 librados: starting msgr at 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 1 librados: starting objecter 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 1 Processor -- start 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 1 -- start start 2024-03-25T19:37:42.123 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 1 librados: setting wanted keys 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 1 librados: calling monclient init 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 10 monclient: init 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.120+0000 7f1e0c515640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.121+0000 7f1e0c515640 10 monclient(hunting): picked mon.noname-b con 0x7f1e040a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.121+0000 7f1e0c515640 10 monclient(hunting): picked mon.noname-c con 0x7f1e0413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.121+0000 7f1e0c515640 10 monclient(hunting): picked mon.noname-a con 0x7f1e041430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.121+0000 7f1e0c515640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e04137df0 con 0x7f1e041430c0 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.121+0000 7f1e0c515640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e041337a0 con 0x7f1e040a98f0 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.121+0000 7f1e0c515640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e04133930 con 0x7f1e0413f710 2024-03-25T19:37:42.124 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.121+0000 7f1e0c515640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:42.125 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.121+0000 7f1e0c515640 10 monclient(hunting): authenticate will time out at 6478.464743s 2024-03-25T19:37:42.125 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.126+0000 7f1e0aab2640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1e041430c0 0x7f1e04133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:53724/0 (socket says 172.21.15.143:53724) 2024-03-25T19:37:42.126 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.126+0000 7f1e0aab2640 1 -- 172.21.15.143:0/1560471213 learned_addr learned my addr 172.21.15.143:0/1560471213 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:42.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.127+0000 7f1e02ffd640 10 client.?.objecter ms_handle_connect 0x7f1e041430c0 2024-03-25T19:37:42.127 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.127+0000 7f1e02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:42.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.134+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553430013 0 0) 0x7f1e04137df0 con 0x7f1e041430c0 2024-03-25T19:37:42.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.134+0000 7f1e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:37:42.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.134+0000 7f1e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:42.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.134+0000 7f1e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:42.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.134+0000 7f1e02ffd640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:37:42.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.134+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df0004eb0 con 0x7f1e041430c0 2024-03-25T19:37:42.138 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.138+0000 7f1e02ffd640 10 client.?.objecter ms_handle_connect 0x7f1e040a98f0 2024-03-25T19:37:42.138 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.138+0000 7f1e02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:42.144 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.144+0000 7f1e02ffd640 10 client.?.objecter ms_handle_connect 0x7f1e0413f710 2024-03-25T19:37:42.144 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.144+0000 7f1e02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:42.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.145+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4047677765 0 0) 0x7f1df0004eb0 con 0x7f1e041430c0 2024-03-25T19:37:42.145 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.145+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e04137df0 con 0x7f1e041430c0 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.146+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1dfc001c70 con 0x7f1e041430c0 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.146+0000 7f1e02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.146+0000 7f1e02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.146+0000 7f1e02ffd640 10 monclient(hunting): dump: 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:42.146 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:42.147 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:42.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.146+0000 7f1e02ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 521732596 0 0) 0x7f1e04137df0 con 0x7f1e041430c0 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 >> v1:172.21.15.143:6790/0 conn(0x7f1e0413f710 legacy=0x7f1e041376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 >> v1:172.21.15.203:6789/0 conn(0x7f1e040a98f0 legacy=0x7f1e041392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 1 monclient: found mon.a 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e041367d0 con 0x7f1e041430c0 2024-03-25T19:37:42.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 10 monclient: _finish_auth 0 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 5 monclient: authenticate success, global_id 4575 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 10 monclient: _renew_subs 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e041369e0 con 0x7f1e041430c0 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 10 monclient: _renew_subs 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0413b640 con 0x7f1e041430c0 2024-03-25T19:37:42.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.151+0000 7f1e0c515640 1 librados: init done 2024-03-25T19:37:42.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.152+0000 7f1e0c515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:42.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.152+0000 7f1e0c515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:42.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.152+0000 7f1e0c515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:42.153 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.152+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e040627b0 con 0x7f1e041430c0 2024-03-25T19:37:42.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.471+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dfc003970 con 0x7f1e041430c0 2024-03-25T19:37:42.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.471+0000 7f1e02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:42.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.471+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1dfc003d80 con 0x7f1e041430c0 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.471+0000 7f1e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.471+0000 7f1e02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.471+0000 7f1e02ffd640 10 monclient: dump: 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:42.472 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:42.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.471+0000 7f1e08aae640 4 set_mon_vals no callback set 2024-03-25T19:37:42.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.472+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f1dfc003fa0 con 0x7f1e041430c0 2024-03-25T19:37:42.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.473+0000 7f1e0a2b1640 10 monclient: get_auth_request con 0x7f1df006dc70 auth_method 0 2024-03-25T19:37:42.473 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.473+0000 7f1e02ffd640 10 client.4575.objecter ms_handle_connect 0x7f1df006dc70 2024-03-25T19:37:42.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.480+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f1dfc054b00 con 0x7f1e041430c0 2024-03-25T19:37:42.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.480+0000 7f1e02ffd640 10 client.4575.objecter ms_dispatch 0x7f1e04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:42.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.480+0000 7f1e02ffd640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:42.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.480+0000 7f1e02ffd640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:42.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.480+0000 7f1e02ffd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T19:37:42.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.480+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dfc050610 con 0x7f1e041430c0 2024-03-25T19:37:42.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.480+0000 7f1e02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:42.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.480+0000 7f1e02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:42.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.724+0000 7f1e0c515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:42.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.724+0000 7f1e0c515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:42.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.724+0000 7f1e0c515640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:42.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.724+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e0414f240 con 0x7f1e041430c0 2024-03-25T19:37:42.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.748+0000 7f1e02ffd640 1 -- 172.21.15.143:0/1560471213 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f1dfc052540 con 0x7f1e041430c0 2024-03-25T19:37:42.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.748+0000 7f1e02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:42.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.748+0000 7f1e02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:42.749 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:37:42.749 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:42.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 10 librados: watch_flush enter 2024-03-25T19:37:42.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 10 librados: watch_flush exit 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 10 client.4575.objecter successfully canceled tick 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 >> v1:172.21.15.143:6800/1292282666 conn(0x7f1df006dc70 legacy=0x7f1df0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 10 monclient: shutdown 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 >> v1:172.21.15.143:6789/0 conn(0x7f1e041430c0 legacy=0x7f1e04133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 shutdown_connections 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.751+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 >> 172.21.15.143:0/1560471213 conn(0x7f1e040b7ea0 msgr2=0x7f1e04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.752+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 shutdown_connections 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.752+0000 7f1e0c515640 1 -- 172.21.15.143:0/1560471213 wait complete. 2024-03-25T19:37:42.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:42.752+0000 7f1e0c515640 1 librados: shutdown 2024-03-25T19:37:42.764 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:49.570 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.680+0000 7f18dc24e640 10 monclient: get_monmap_and_config 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.680+0000 7f18dc24e640 10 monclient: build_initial_monmap 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.680+0000 7f18dc24e640 10 monclient: monmap: 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:37:49.681339+0000 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:37:49.681339+0000 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:49.680 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:49.681 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:49.681 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:37:49.681 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:37:49.681 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:49.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.681+0000 7f18dc24e640 1 Processor -- start 2024-03-25T19:37:49.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.681+0000 7f18dc24e640 1 -- start start 2024-03-25T19:37:49.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.681+0000 7f18dc24e640 10 monclient: init 2024-03-25T19:37:49.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:49.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:37:49.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 10 monclient(hunting): picked mon.noname-c con 0x7f18d40a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:49.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 10 monclient(hunting): picked mon.noname-b con 0x7f18d413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:49.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 10 monclient(hunting): picked mon.noname-a con 0x7f18d41430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:49.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d4142520 con 0x7f18d41430c0 2024-03-25T19:37:49.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d40b28a0 con 0x7f18d40a98f0 2024-03-25T19:37:49.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d40b2a30 con 0x7f18d413f710 2024-03-25T19:37:49.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:49.683 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.682+0000 7f18dc24e640 10 monclient(hunting): authenticate will time out at 6486.025986s 2024-03-25T19:37:49.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.693+0000 7f18d9fea640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f18d40a98f0 0x7f18d40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:34832/0 (socket says 172.21.15.143:34832) 2024-03-25T19:37:49.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.693+0000 7f18d9fea640 1 -- 172.21.15.143:0/3329941456 learned_addr learned my addr 172.21.15.143:0/3329941456 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:49.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.702+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577359962 0 0) 0x7f18d40b28a0 con 0x7f18d40a98f0 2024-03-25T19:37:49.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.702+0000 7f18d8fe8640 10 monclient(hunting): _negotiate 2024-03-25T19:37:49.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.702+0000 7f18d8fe8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:49.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.702+0000 7f18d8fe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:49.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.702+0000 7f18d8fe8640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:37:49.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.702+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c0003520 con 0x7f18d40a98f0 2024-03-25T19:37:49.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 318915387 0 0) 0x7f18c0003520 con 0x7f18d40a98f0 2024-03-25T19:37:49.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 >> v1:172.21.15.143:6790/0 conn(0x7f18d413f710 legacy=0x7f18d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:49.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 >> v1:172.21.15.143:6789/0 conn(0x7f18d41430c0 legacy=0x7f18d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:49.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 10 monclient: _finish_hunting 0 2024-03-25T19:37:49.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 1 monclient: found mon.noname-c 2024-03-25T19:37:49.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d40b44c0 con 0x7f18d40a98f0 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 10 monclient: _finish_auth 0 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 10 monclient: _check_auth_tickets 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f18d0001690 con 0x7f18d40a98f0 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18d8fe8640 10 monclient: dump: 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:49.707 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:49.708 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:49.708 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:49.708 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:49.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18dc24e640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:37:49.708 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.706+0000 7f18dc24e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:49.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d00017f0 con 0x7f18d40a98f0 2024-03-25T19:37:49.717 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18d8fe8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18d8fe8640 1 -- 172.21.15.143:0/3329941456 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f18d0001c00 con 0x7f18d40a98f0 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18d8fe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18d8fe8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18d8fe8640 10 monclient: dump: 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:49.718 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:49.719 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:49.719 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:49.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18dc24e640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:49.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18dc24e640 4 set_mon_vals no callback set 2024-03-25T19:37:49.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18dc24e640 10 monclient: shutdown 2024-03-25T19:37:49.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18dc24e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:49.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.718+0000 7f18dc24e640 1 -- 172.21.15.143:0/3329941456 >> v1:172.21.15.203:6789/0 conn(0x7f18d40a98f0 legacy=0x7f18d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.719+0000 7f18dc24e640 1 -- 172.21.15.143:0/3329941456 shutdown_connections 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.719+0000 7f18dc24e640 1 -- 172.21.15.143:0/3329941456 >> 172.21.15.143:0/3329941456 conn(0x7f18d40b7ea0 msgr2=0x7f18d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.719+0000 7f18dc24e640 1 -- 172.21.15.143:0/3329941456 shutdown_connections 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.719+0000 7f18dc24e640 1 -- 172.21.15.143:0/3329941456 wait complete. 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.719+0000 7f18dc24e640 10 monclient: build_initial_monmap 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.719+0000 7f18dc24e640 10 monclient: monmap: 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:49.721 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 1 librados: starting msgr at 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 1 librados: starting objecter 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 1 Processor -- start 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 1 -- start start 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 1 librados: setting wanted keys 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 1 librados: calling monclient init 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 10 monclient: init 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 10 monclient(hunting): picked mon.noname-a con 0x7f18d40a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 10 monclient(hunting): picked mon.noname-b con 0x7f18d413f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.720+0000 7f18dc24e640 10 monclient(hunting): picked mon.noname-c con 0x7f18d41430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:49.722 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.721+0000 7f18dc24e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d4138b20 con 0x7f18d40a98f0 2024-03-25T19:37:49.723 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.721+0000 7f18dc24e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d4137a80 con 0x7f18d413f710 2024-03-25T19:37:49.723 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.721+0000 7f18dc24e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d4139880 con 0x7f18d41430c0 2024-03-25T19:37:49.723 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.721+0000 7f18dc24e640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:49.723 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.721+0000 7f18dc24e640 10 monclient(hunting): authenticate will time out at 6486.064690s 2024-03-25T19:37:49.740 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.740+0000 7f18d97e9640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f18d413f710 0x7f18d4139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:34884/0 (socket says 172.21.15.143:34884) 2024-03-25T19:37:49.740 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.741+0000 7f18d97e9640 1 -- 172.21.15.143:0/3188287579 learned_addr learned my addr 172.21.15.143:0/3188287579 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:49.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.744+0000 7f18ca7fc640 10 client.?.objecter ms_handle_connect 0x7f18d413f710 2024-03-25T19:37:49.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.744+0000 7f18ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:49.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.744+0000 7f18ca7fc640 10 client.?.objecter ms_handle_connect 0x7f18d40a98f0 2024-03-25T19:37:49.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.744+0000 7f18ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:49.749 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.750+0000 7f18ca7fc640 10 client.?.objecter ms_handle_connect 0x7f18d41430c0 2024-03-25T19:37:49.749 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.750+0000 7f18ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:49.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.753+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074059606 0 0) 0x7f18d4138b20 con 0x7f18d40a98f0 2024-03-25T19:37:49.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.753+0000 7f18ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:49.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.753+0000 7f18ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:49.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.753+0000 7f18ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:49.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.753+0000 7f18ca7fc640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:37:49.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.753+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18b8004eb0 con 0x7f18d40a98f0 2024-03-25T19:37:49.755 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.755+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122665776 0 0) 0x7f18d4137a80 con 0x7f18d413f710 2024-03-25T19:37:49.755 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.755+0000 7f18ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:49.755 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.755+0000 7f18ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:49.755 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.755+0000 7f18ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:49.755 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.755+0000 7f18ca7fc640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:37:49.755 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.755+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d4138b20 con 0x7f18d413f710 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.768+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250615094 0 0) 0x7f18d4139880 con 0x7f18d41430c0 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.768+0000 7f18ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.768+0000 7f18ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.768+0000 7f18ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.768+0000 7f18ca7fc640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.768+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d4137a80 con 0x7f18d41430c0 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.769+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4011746370 0 0) 0x7f18d4138b20 con 0x7f18d413f710 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.769+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18d4139880 con 0x7f18d413f710 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.769+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f18c4001970 con 0x7f18d413f710 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.769+0000 7f18ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:49.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.769+0000 7f18ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.769+0000 7f18ca7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:49.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.769+0000 7f18ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:49.772 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.773+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413571703 0 0) 0x7f18b8004eb0 con 0x7f18d40a98f0 2024-03-25T19:37:49.772 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.773+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18d4138b20 con 0x7f18d40a98f0 2024-03-25T19:37:49.772 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.773+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f18d0001bc0 con 0x7f18d40a98f0 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.773+0000 7f18ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.773+0000 7f18ca7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.773+0000 7f18ca7fc640 10 monclient(hunting): dump: 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:49.773 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:49.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3518446389 0 0) 0x7f18d4138b20 con 0x7f18d40a98f0 2024-03-25T19:37:49.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 >> v1:172.21.15.143:6790/0 conn(0x7f18d41430c0 legacy=0x7f18d41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:49.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 >> v1:172.21.15.203:6789/0 conn(0x7f18d413f710 legacy=0x7f18d4139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:49.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 1 monclient: found mon.a 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4139a10 con 0x7f18d40a98f0 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 5 monclient: authenticate success, global_id 4578 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 10 monclient: _renew_subs 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d41dab00 con 0x7f18d40a98f0 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 10 monclient: _renew_subs 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:49.779 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d413b4f0 con 0x7f18d40a98f0 2024-03-25T19:37:49.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.778+0000 7f18dc24e640 1 librados: init done 2024-03-25T19:37:49.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.779+0000 7f18dc24e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:49.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.779+0000 7f18dc24e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:49.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.779+0000 7f18dc24e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:49.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.779+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d40b3e80 con 0x7f18d40a98f0 2024-03-25T19:37:49.805 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.805+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d0004a90 con 0x7f18d40a98f0 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.806+0000 7f18ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.806+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f18d0001250 con 0x7f18d40a98f0 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.806+0000 7f18ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.806+0000 7f18cbfff640 4 set_mon_vals no callback set 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.806+0000 7f18ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.806+0000 7f18ca7fc640 10 monclient: dump: 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.806 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:49.807 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:49.807 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:49.807 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:49.807 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:49.807 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:49.807 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:49.807 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.807+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f18d001ac50 con 0x7f18d40a98f0 2024-03-25T19:37:49.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.807+0000 7f18d97e9640 10 monclient: get_auth_request con 0x7f18b806df70 auth_method 0 2024-03-25T19:37:49.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:49.808+0000 7f18ca7fc640 10 client.4578.objecter ms_handle_connect 0x7f18b806df70 2024-03-25T19:37:50.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.096+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f18d0054900 con 0x7f18d40a98f0 2024-03-25T19:37:50.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.096+0000 7f18ca7fc640 10 client.4578.objecter ms_dispatch 0x7f18d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:50.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.096+0000 7f18ca7fc640 3 client.4578.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:50.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.096+0000 7f18ca7fc640 3 client.4578.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:50.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.096+0000 7f18ca7fc640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-25T19:37:50.096 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.096+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18d0050240 con 0x7f18d40a98f0 2024-03-25T19:37:50.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.096+0000 7f18ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:50.097 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.096+0000 7f18ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:50.354 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.355+0000 7f18dc24e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:50.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.355+0000 7f18dc24e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:50.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.355+0000 7f18dc24e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:50.355 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.355+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18d40627b0 con 0x7f18d40a98f0 2024-03-25T19:37:50.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.379+0000 7f18ca7fc640 1 -- 172.21.15.143:0/3188287579 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f18d0052340 con 0x7f18d40a98f0 2024-03-25T19:37:50.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.380+0000 7f18ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:50.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.380+0000 7f18ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:50.380 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:37:50.380 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:50.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 10 librados: watch_flush enter 2024-03-25T19:37:50.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 10 librados: watch_flush exit 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 10 client.4578.objecter successfully canceled tick 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 >> v1:172.21.15.143:6800/1292282666 conn(0x7f18b806df70 legacy=0x7f18b8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 10 monclient: shutdown 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 >> v1:172.21.15.143:6789/0 conn(0x7f18d40a98f0 legacy=0x7f18d4137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 shutdown_connections 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 >> 172.21.15.143:0/3188287579 conn(0x7f18d40b7ea0 msgr2=0x7f18d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 shutdown_connections 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 1 -- 172.21.15.143:0/3188287579 wait complete. 2024-03-25T19:37:50.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:50.383+0000 7f18dc24e640 1 librados: shutdown 2024-03-25T19:37:50.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:57.201 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.321+0000 7f7ffda4a640 10 monclient: get_monmap_and_config 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.321+0000 7f7ffda4a640 10 monclient: build_initial_monmap 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.321+0000 7f7ffda4a640 10 monclient: monmap: 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:37:57.322487+0000 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:37:57.322487+0000 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:57.321 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:37:57.322 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:37:57.322 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.322+0000 7f7ffda4a640 1 Processor -- start 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.322+0000 7f7ffda4a640 1 -- start start 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.322+0000 7f7ffda4a640 10 monclient: init 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff813f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff81430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8142520 con 0x7f7ff81430c0 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80b28a0 con 0x7f7ff813f710 2024-03-25T19:37:57.323 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80b2a30 con 0x7f7ff80a98f0 2024-03-25T19:37:57.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:57.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.323+0000 7f7ffda4a640 10 monclient(hunting): authenticate will time out at 6493.667166s 2024-03-25T19:37:57.333 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.333+0000 7f7ff67fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7ff813f710 0x7f7ff81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:49060/0 (socket says 172.21.15.143:49060) 2024-03-25T19:37:57.333 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.333+0000 7f7ff67fc640 1 -- 172.21.15.143:0/736543213 learned_addr learned my addr 172.21.15.143:0/736543213 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:57.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.348+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435141985 0 0) 0x7f7ff80b28a0 con 0x7f7ff813f710 2024-03-25T19:37:57.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.348+0000 7f7ff5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:57.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.348+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:57.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.348+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:57.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.348+0000 7f7ff5ffb640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:37:57.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.348+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0003520 con 0x7f7ff813f710 2024-03-25T19:37:57.352 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.353+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292403266 0 0) 0x7f7ff8142520 con 0x7f7ff81430c0 2024-03-25T19:37:57.352 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.353+0000 7f7ff5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:57.352 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.353+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:57.352 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.353+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:57.352 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.353+0000 7f7ff5ffb640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:37:57.353 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.353+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff80b28a0 con 0x7f7ff81430c0 2024-03-25T19:37:57.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.359+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562762948 0 0) 0x7f7ff80b2a30 con 0x7f7ff80a98f0 2024-03-25T19:37:57.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.359+0000 7f7ff5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:37:57.358 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.359+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:57.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.359+0000 7f7ff5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:57.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.359+0000 7f7ff5ffb640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:37:57.359 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.359+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8142520 con 0x7f7ff80a98f0 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888373831 0 0) 0x7f7fe0003520 con 0x7f7ff813f710 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 >> v1:172.21.15.143:6790/0 conn(0x7f7ff80a98f0 legacy=0x7f7ff80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 >> v1:172.21.15.143:6789/0 conn(0x7f7ff81430c0 legacy=0x7f7ff814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff80b44c0 con 0x7f7ff813f710 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:37:57.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7fec001690 con 0x7f7ff813f710 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ff5ffb640 10 monclient: dump: 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:57.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ffda4a640 5 monclient: authenticate success, global_id 4585 2024-03-25T19:37:57.370 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.368+0000 7f7ffda4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:37:57.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.380+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fec0017f0 con 0x7f7ff813f710 2024-03-25T19:37:57.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.380+0000 7f7ff5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.380+0000 7f7ff5ffb640 1 -- 172.21.15.143:0/736543213 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7fec001c00 con 0x7f7ff813f710 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.380+0000 7f7ff5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.380+0000 7f7ff5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.380+0000 7f7ff5ffb640 10 monclient: dump: 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:57.380 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:57.381 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:57.381 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:57.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.380+0000 7f7ffda4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:37:57.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.380+0000 7f7ffda4a640 4 set_mon_vals no callback set 2024-03-25T19:37:57.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 10 monclient: shutdown 2024-03-25T19:37:57.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:57.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 1 -- 172.21.15.143:0/736543213 >> v1:172.21.15.203:6789/0 conn(0x7f7ff813f710 legacy=0x7f7ff81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:57.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 1 -- 172.21.15.143:0/736543213 shutdown_connections 2024-03-25T19:37:57.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 1 -- 172.21.15.143:0/736543213 >> 172.21.15.143:0/736543213 conn(0x7f7ff80b7ea0 msgr2=0x7f7ff80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:57.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 1 -- 172.21.15.143:0/736543213 shutdown_connections 2024-03-25T19:37:57.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 1 -- 172.21.15.143:0/736543213 wait complete. 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 10 monclient: build_initial_monmap 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.381+0000 7f7ffda4a640 10 monclient: monmap: 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.382+0000 7f7ffda4a640 1 librados: starting msgr at 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.382+0000 7f7ffda4a640 1 librados: starting objecter 2024-03-25T19:37:57.384 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.382+0000 7f7ffda4a640 1 Processor -- start 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.382+0000 7f7ffda4a640 1 -- start start 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.382+0000 7f7ffda4a640 1 librados: setting wanted keys 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.382+0000 7f7ffda4a640 1 librados: calling monclient init 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.382+0000 7f7ffda4a640 10 monclient: init 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.382+0000 7f7ffda4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff80a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff813f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff81430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff81d77a0 con 0x7f7ff81430c0 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8138e40 con 0x7f7ff813f710 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80aba30 con 0x7f7ff80a98f0 2024-03-25T19:37:57.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:37:57.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.383+0000 7f7ffda4a640 10 monclient(hunting): authenticate will time out at 6493.726829s 2024-03-25T19:37:57.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.393+0000 7f7ff67fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7ff813f710 0x7f7ff81d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:49080/0 (socket says 172.21.15.143:49080) 2024-03-25T19:37:57.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.393+0000 7f7ff67fc640 1 -- 172.21.15.143:0/4276506186 learned_addr learned my addr 172.21.15.143:0/4276506186 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:37:57.394 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.395+0000 7f7fdffff640 10 client.?.objecter ms_handle_connect 0x7f7ff813f710 2024-03-25T19:37:57.394 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.395+0000 7f7fdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:57.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.395+0000 7f7fdffff640 10 client.?.objecter ms_handle_connect 0x7f7ff80a98f0 2024-03-25T19:37:57.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.395+0000 7f7fdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:57.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.396+0000 7f7fdffff640 10 client.?.objecter ms_handle_connect 0x7f7ff81430c0 2024-03-25T19:37:57.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.396+0000 7f7fdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:37:57.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.400+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224653209 0 0) 0x7f7ff80aba30 con 0x7f7ff80a98f0 2024-03-25T19:37:57.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.400+0000 7f7fdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:57.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.400+0000 7f7fdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:57.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.400+0000 7f7fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:57.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.400+0000 7f7fdffff640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:37:57.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.400+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff80a98f0 2024-03-25T19:37:57.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.403+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190299027 0 0) 0x7f7ff81d77a0 con 0x7f7ff81430c0 2024-03-25T19:37:57.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.403+0000 7f7fdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:57.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.403+0000 7f7fdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:57.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.403+0000 7f7fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:57.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.403+0000 7f7fdffff640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:37:57.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.403+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff80aba30 con 0x7f7ff81430c0 2024-03-25T19:37:57.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.404+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072398130 0 0) 0x7f7ff8138e40 con 0x7f7ff813f710 2024-03-25T19:37:57.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.404+0000 7f7fdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:37:57.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.404+0000 7f7fdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:37:57.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.404+0000 7f7fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:37:57.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.404+0000 7f7fdffff640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:37:57.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.404+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff81d77a0 con 0x7f7ff813f710 2024-03-25T19:37:57.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.413+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262211220 0 0) 0x7f7fe4004eb0 con 0x7f7ff80a98f0 2024-03-25T19:37:57.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff8138e40 con 0x7f7ff80a98f0 2024-03-25T19:37:57.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7fe80018f0 con 0x7f7ff80a98f0 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 10 monclient(hunting): dump: 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:57.414 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:37:57.415 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 896119650 0 0) 0x7f7ff80aba30 con 0x7f7ff81430c0 2024-03-25T19:37:57.415 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff81430c0 2024-03-25T19:37:57.415 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7ff0001bd0 con 0x7f7ff81430c0 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.414+0000 7f7fdffff640 10 monclient(hunting): dump: 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.415+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3966229089 0 0) 0x7f7ff81d77a0 con 0x7f7ff813f710 2024-03-25T19:37:57.416 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.415+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff80aba30 con 0x7f7ff813f710 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.415+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7fec001b00 con 0x7f7ff813f710 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.415+0000 7f7fdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.415+0000 7f7fdffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.415+0000 7f7fdffff640 10 monclient(hunting): dump: 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:57.417 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 310791013 0 0) 0x7f7fe4004eb0 con 0x7f7ff81430c0 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 >> v1:172.21.15.143:6790/0 conn(0x7f7ff80a98f0 legacy=0x7f7ff8138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 >> v1:172.21.15.203:6789/0 conn(0x7f7ff813f710 legacy=0x7f7ff81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 10 monclient: _finish_hunting 0 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 1 monclient: found mon.a 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff81d8340 con 0x7f7ff81430c0 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 10 monclient: _finish_auth 0 2024-03-25T19:37:57.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 10 monclient: _check_auth_tickets 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7fdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 5 monclient: authenticate success, global_id 4584 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 10 monclient: _renew_subs 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff8134820 con 0x7f7ff81430c0 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 10 monclient: _renew_subs 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff81380c0 con 0x7f7ff81430c0 2024-03-25T19:37:57.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.424+0000 7f7ffda4a640 1 librados: init done 2024-03-25T19:37:57.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.425+0000 7f7ffda4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:57.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.425+0000 7f7ffda4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:57.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.425+0000 7f7ffda4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:57.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.425+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff80627b0 con 0x7f7ff81430c0 2024-03-25T19:37:57.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.718+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ff0002060 con 0x7f7ff81430c0 2024-03-25T19:37:57.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.718+0000 7f7fdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:37:57.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.718+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7ff0003b10 con 0x7f7ff81430c0 2024-03-25T19:37:57.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.718+0000 7f7fdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:37:57.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.718+0000 7f7fdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:37:57.718 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.718+0000 7f7fdffff640 10 monclient: dump: 2024-03-25T19:37:57.718 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:37:57.718 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:37:57.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.718+0000 7f7ff57fa640 4 set_mon_vals no callback set 2024-03-25T19:37:57.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.719+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f7ff0019ae0 con 0x7f7ff81430c0 2024-03-25T19:37:57.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.719+0000 7f7ff6ffd640 10 monclient: get_auth_request con 0x7f7fe406e080 auth_method 0 2024-03-25T19:37:57.720 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.719+0000 7f7fdffff640 10 client.4584.objecter ms_handle_connect 0x7f7fe406e080 2024-03-25T19:37:57.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.724+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f7ff0053880 con 0x7f7ff81430c0 2024-03-25T19:37:57.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.724+0000 7f7fdffff640 10 client.4584.objecter ms_dispatch 0x7f7ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:37:57.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.724+0000 7f7fdffff640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:37:57.724 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.724+0000 7f7fdffff640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:37:57.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.724+0000 7f7fdffff640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-25T19:37:57.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.724+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ff004f1c0 con 0x7f7ff81430c0 2024-03-25T19:37:57.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.724+0000 7f7fdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:37:57.725 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.724+0000 7f7fdffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:37:57.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.957+0000 7f7ffda4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:57.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.957+0000 7f7ffda4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:57.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.957+0000 7f7ffda4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:37:57.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.957+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ff814e880 con 0x7f7ff81430c0 2024-03-25T19:37:57.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.980+0000 7f7fdffff640 1 -- 172.21.15.143:0/4276506186 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f7ff00512c0 con 0x7f7ff81430c0 2024-03-25T19:37:57.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.980+0000 7f7fdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:37:57.980 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.980+0000 7f7fdffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:37:57.980 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:37:57.981 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:37:57.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 10 librados: watch_flush enter 2024-03-25T19:37:57.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 10 librados: watch_flush exit 2024-03-25T19:37:57.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-25T19:37:57.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 10 client.4584.objecter successfully canceled tick 2024-03-25T19:37:57.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 >> v1:172.21.15.143:6800/1292282666 conn(0x7f7fe406e080 legacy=0x7f7fe4070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:57.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 10 monclient: shutdown 2024-03-25T19:37:57.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:37:57.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 >> v1:172.21.15.143:6789/0 conn(0x7f7ff81430c0 legacy=0x7f7ff8134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:37:57.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 shutdown_connections 2024-03-25T19:37:57.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 >> 172.21.15.143:0/4276506186 conn(0x7f7ff80b7ea0 msgr2=0x7f7ff8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:37:57.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 shutdown_connections 2024-03-25T19:37:57.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 1 -- 172.21.15.143:0/4276506186 wait complete. 2024-03-25T19:37:57.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:37:57.983+0000 7f7ffda4a640 1 librados: shutdown 2024-03-25T19:37:57.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:37:59.540 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:26.104 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.553 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:26.118 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.592 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:26.157 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.594 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:26.159 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.603 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:26.138 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.606 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:26.171 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.620 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:26.155 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.638 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:26.203 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:37:59.643 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:26.179 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:04.799 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:04.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.904+0000 7efca9886640 10 monclient: get_monmap_and_config 2024-03-25T19:38:04.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.904+0000 7efca9886640 10 monclient: build_initial_monmap 2024-03-25T19:38:04.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.904+0000 7efca9886640 10 monclient: monmap: 2024-03-25T19:38:04.905 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:38:04.906805+0000 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:38:04.906805+0000 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:38:04.906 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:04.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.905+0000 7efca9886640 1 Processor -- start 2024-03-25T19:38:04.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.905+0000 7efca9886640 1 -- start start 2024-03-25T19:38:04.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.905+0000 7efca9886640 10 monclient: init 2024-03-25T19:38:04.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 10 monclient(hunting): picked mon.noname-a con 0x7efca40a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 10 monclient(hunting): picked mon.noname-c con 0x7efca413f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 10 monclient(hunting): picked mon.noname-b con 0x7efca41430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca4142520 con 0x7efca40a98f0 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca40b28a0 con 0x7efca413f710 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca40b2a30 con 0x7efca41430c0 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:04.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.906+0000 7efca9886640 10 monclient(hunting): authenticate will time out at 6501.250900s 2024-03-25T19:38:04.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.918+0000 7efca27fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7efca413f710 0x7efca41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:53596/0 (socket says 172.21.15.143:53596) 2024-03-25T19:38:04.918 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.918+0000 7efca27fc640 1 -- 172.21.15.143:0/1891294337 learned_addr learned my addr 172.21.15.143:0/1891294337 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:04.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.927+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398085528 0 0) 0x7efca40b28a0 con 0x7efca413f710 2024-03-25T19:38:04.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.927+0000 7efca1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:04.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.927+0000 7efca1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:04.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.927+0000 7efca1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:04.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.927+0000 7efca1ffb640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:38:04.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.927+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8c003520 con 0x7efca413f710 2024-03-25T19:38:04.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.929+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826613336 0 0) 0x7efca40b2a30 con 0x7efca41430c0 2024-03-25T19:38:04.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.930+0000 7efca1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:04.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.930+0000 7efca1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:04.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.930+0000 7efca1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:04.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.930+0000 7efca1ffb640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:38:04.930 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.930+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca40b28a0 con 0x7efca41430c0 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3567215365 0 0) 0x7efc8c003520 con 0x7efca413f710 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 >> v1:172.21.15.143:6790/0 conn(0x7efca41430c0 legacy=0x7efca414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 >> v1:172.21.15.143:6789/0 conn(0x7efca40a98f0 legacy=0x7efca40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 1 monclient: found mon.noname-c 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca40b44c0 con 0x7efca413f710 2024-03-25T19:38:04.940 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efc98001690 con 0x7efca413f710 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca1ffb640 10 monclient: dump: 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:04.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca9886640 5 monclient: authenticate success, global_id 4591 2024-03-25T19:38:04.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.939+0000 7efca9886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.967+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc980017f0 con 0x7efca413f710 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.967+0000 7efca1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.967+0000 7efca1ffb640 1 -- 172.21.15.143:0/1891294337 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efc98001c00 con 0x7efca413f710 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.967+0000 7efca1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.967+0000 7efca1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.967+0000 7efca1ffb640 10 monclient: dump: 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:04.968 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.967+0000 7efca9886640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:04.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.967+0000 7efca9886640 4 set_mon_vals no callback set 2024-03-25T19:38:04.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.968+0000 7efca9886640 10 monclient: shutdown 2024-03-25T19:38:04.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.968+0000 7efca9886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:04.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.968+0000 7efca9886640 1 -- 172.21.15.143:0/1891294337 >> v1:172.21.15.203:6789/0 conn(0x7efca413f710 legacy=0x7efca41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:04.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.968+0000 7efca9886640 1 -- 172.21.15.143:0/1891294337 shutdown_connections 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.968+0000 7efca9886640 1 -- 172.21.15.143:0/1891294337 >> 172.21.15.143:0/1891294337 conn(0x7efca40b7ea0 msgr2=0x7efca40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.968+0000 7efca9886640 1 -- 172.21.15.143:0/1891294337 shutdown_connections 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.968+0000 7efca9886640 1 -- 172.21.15.143:0/1891294337 wait complete. 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.969+0000 7efca9886640 10 monclient: build_initial_monmap 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.969+0000 7efca9886640 10 monclient: monmap: 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.969+0000 7efca9886640 1 librados: starting msgr at 2024-03-25T19:38:04.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.969+0000 7efca9886640 1 librados: starting objecter 2024-03-25T19:38:04.972 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.969+0000 7efca9886640 1 Processor -- start 2024-03-25T19:38:04.972 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.970+0000 7efca9886640 1 -- start start 2024-03-25T19:38:04.972 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.970+0000 7efca9886640 1 librados: setting wanted keys 2024-03-25T19:38:04.972 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.970+0000 7efca9886640 1 librados: calling monclient init 2024-03-25T19:38:04.972 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.970+0000 7efca9886640 10 monclient: init 2024-03-25T19:38:04.972 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.970+0000 7efca9886640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.970+0000 7efca9886640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.971+0000 7efca9886640 10 monclient(hunting): picked mon.noname-a con 0x7efca40a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.971+0000 7efca9886640 10 monclient(hunting): picked mon.noname-c con 0x7efca413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.971+0000 7efca9886640 10 monclient(hunting): picked mon.noname-b con 0x7efca41430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.971+0000 7efca9886640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca41376a0 con 0x7efca40a98f0 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.971+0000 7efca9886640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca4131870 con 0x7efca41430c0 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.971+0000 7efca9886640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca4138e10 con 0x7efca413f710 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.971+0000 7efca9886640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:04.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.971+0000 7efca9886640 10 monclient(hunting): authenticate will time out at 6501.315894s 2024-03-25T19:38:04.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.977+0000 7efca27fc640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7efca413f710 0x7efca4130a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:35524/0 (socket says 172.21.15.143:35524) 2024-03-25T19:38:04.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.977+0000 7efca27fc640 1 -- 172.21.15.143:0/635767348 learned_addr learned my addr 172.21.15.143:0/635767348 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:04.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.978+0000 7efc8bfff640 10 client.?.objecter ms_handle_connect 0x7efca413f710 2024-03-25T19:38:04.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.978+0000 7efc8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:04.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.983+0000 7efc8bfff640 10 client.?.objecter ms_handle_connect 0x7efca40a98f0 2024-03-25T19:38:04.984 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.983+0000 7efc8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:04.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.990+0000 7efc8bfff640 10 client.?.objecter ms_handle_connect 0x7efca41430c0 2024-03-25T19:38:04.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.990+0000 7efc8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:04.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.992+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497803998 0 0) 0x7efca4138e10 con 0x7efca413f710 2024-03-25T19:38:04.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.992+0000 7efc8bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:04.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.992+0000 7efc8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:04.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.992+0000 7efc8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:04.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.992+0000 7efc8bfff640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:38:04.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.992+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc90004eb0 con 0x7efca413f710 2024-03-25T19:38:05.000 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.999+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990379141 0 0) 0x7efca4131870 con 0x7efca41430c0 2024-03-25T19:38:05.000 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.999+0000 7efc8bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:05.000 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.999+0000 7efc8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:05.000 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.999+0000 7efc8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:05.000 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.999+0000 7efc8bfff640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:38:05.001 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:04.999+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca4138e10 con 0x7efca41430c0 2024-03-25T19:38:05.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.002+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552790247 0 0) 0x7efca41376a0 con 0x7efca40a98f0 2024-03-25T19:38:05.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.002+0000 7efc8bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:05.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.002+0000 7efc8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:05.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.002+0000 7efc8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:05.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.002+0000 7efc8bfff640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:38:05.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.002+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca4131870 con 0x7efca40a98f0 2024-03-25T19:38:05.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.013+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4079845688 0 0) 0x7efc90004eb0 con 0x7efca413f710 2024-03-25T19:38:05.014 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efca41376a0 con 0x7efca413f710 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efc98001bc0 con 0x7efca413f710 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 10 monclient(hunting): dump: 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:05.015 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:05.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 733920468 0 0) 0x7efca4138e10 con 0x7efca41430c0 2024-03-25T19:38:05.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc90004eb0 con 0x7efca41430c0 2024-03-25T19:38:05.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efc9c001b90 con 0x7efca41430c0 2024-03-25T19:38:05.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:05.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:05.016 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.014+0000 7efc8bfff640 10 monclient(hunting): dump: 2024-03-25T19:38:05.016 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:05.016 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:05.017 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:05.017 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:05.017 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:05.017 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:05.017 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:05.017 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:05.017 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:05.017 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:05.028 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1967716664 0 0) 0x7efc90004eb0 con 0x7efca41430c0 2024-03-25T19:38:05.028 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 >> v1:172.21.15.143:6790/0 conn(0x7efca413f710 legacy=0x7efca4130a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:05.028 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 >> v1:172.21.15.143:6789/0 conn(0x7efca40a98f0 legacy=0x7efca4137b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 1 monclient: found mon.b 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca41390f0 con 0x7efca41430c0 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 10 monclient: _finish_auth 0 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efc8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 5 monclient: authenticate success, global_id 4594 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 10 monclient: _renew_subs 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca41393d0 con 0x7efca41430c0 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 10 monclient: _renew_subs 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:05.029 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efca41397f0 con 0x7efca41430c0 2024-03-25T19:38:05.030 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.027+0000 7efca9886640 1 librados: init done 2024-03-25T19:38:05.030 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.028+0000 7efca9886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:05.030 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.028+0000 7efca9886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:05.030 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.028+0000 7efca9886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:05.030 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.028+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca413d010 con 0x7efca41430c0 2024-03-25T19:38:05.320 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.319+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc9c002040 con 0x7efca41430c0 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.319+0000 7efc8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.319+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7efc9c003b30 con 0x7efca41430c0 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.319+0000 7efc8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.319+0000 7efc8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.319+0000 7efc8bfff640 10 monclient: dump: 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:05.321 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:05.322 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.319+0000 7efca17fa640 4 set_mon_vals no callback set 2024-03-25T19:38:05.322 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.320+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7efc9c019b00 con 0x7efca41430c0 2024-03-25T19:38:05.322 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.321+0000 7efca2ffd640 10 monclient: get_auth_request con 0x7efc9006dfb0 auth_method 0 2024-03-25T19:38:05.322 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.321+0000 7efc8bfff640 10 client.4594.objecter ms_handle_connect 0x7efc9006dfb0 2024-03-25T19:38:05.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.326+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7efc9c053880 con 0x7efca41430c0 2024-03-25T19:38:05.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.326+0000 7efc8bfff640 10 client.4594.objecter ms_dispatch 0x7efca4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:05.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.326+0000 7efc8bfff640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:05.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.326+0000 7efc8bfff640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:05.327 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.326+0000 7efc8bfff640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T19:38:05.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.327+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc9c04f1c0 con 0x7efca41430c0 2024-03-25T19:38:05.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.327+0000 7efc8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:05.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.327+0000 7efc8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:05.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.582+0000 7efca9886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:05.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.583+0000 7efca9886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:05.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.583+0000 7efca9886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:05.584 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.583+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efca4137450 con 0x7efca41430c0 2024-03-25T19:38:05.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.613+0000 7efc8bfff640 1 -- 172.21.15.143:0/635767348 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7efc9c0512c0 con 0x7efca41430c0 2024-03-25T19:38:05.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.613+0000 7efc8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:05.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.613+0000 7efc8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:05.614 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:38:05.614 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:05.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 10 librados: watch_flush enter 2024-03-25T19:38:05.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 10 librados: watch_flush exit 2024-03-25T19:38:05.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T19:38:05.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 10 client.4594.objecter successfully canceled tick 2024-03-25T19:38:05.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 >> v1:172.21.15.143:6800/1292282666 conn(0x7efc9006dfb0 legacy=0x7efc90070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:05.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 10 monclient: shutdown 2024-03-25T19:38:05.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:05.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 >> v1:172.21.15.203:6789/0 conn(0x7efca41430c0 legacy=0x7efca4131160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:05.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 shutdown_connections 2024-03-25T19:38:05.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 >> 172.21.15.143:0/635767348 conn(0x7efca40b7ea0 msgr2=0x7efca4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:05.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 shutdown_connections 2024-03-25T19:38:05.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 1 -- 172.21.15.143:0/635767348 wait complete. 2024-03-25T19:38:05.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:05.616+0000 7efca9886640 1 librados: shutdown 2024-03-25T19:38:05.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:12.436 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:12.545 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.545+0000 7f0796cb3640 10 monclient: get_monmap_and_config 2024-03-25T19:38:12.545 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.545+0000 7f0796cb3640 10 monclient: build_initial_monmap 2024-03-25T19:38:12.545 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.545+0000 7f0796cb3640 10 monclient: monmap: 2024-03-25T19:38:12.545 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:12.545 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:12.545 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:38:12.546540+0000 2024-03-25T19:38:12.546 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:38:12.546540+0000 2024-03-25T19:38:12.546 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:12.546 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:12.546 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:12.546 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:38:12.546 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:38:12.546 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:12.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.546+0000 7f0796cb3640 1 Processor -- start 2024-03-25T19:38:12.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.546+0000 7f0796cb3640 1 -- start start 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.546+0000 7f0796cb3640 10 monclient: init 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f07900b10d0 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f07900b4360 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f07900a9470 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07900a9ab0 con 0x7f07900a9470 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07900b2400 con 0x7f07900b10d0 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07900aa2a0 con 0x7f07900b4360 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:12.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.547+0000 7f0796cb3640 10 monclient(hunting): authenticate will time out at 6508.891353s 2024-03-25T19:38:12.560 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.560+0000 7f0795250640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f07900a9470 0x7f079014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:41742/0 (socket says 172.21.15.143:41742) 2024-03-25T19:38:12.560 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.560+0000 7f0795250640 1 -- 172.21.15.143:0/3015107699 learned_addr learned my addr 172.21.15.143:0/3015107699 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:12.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.579+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504790324 0 0) 0x7f07900a9ab0 con 0x7f07900a9470 2024-03-25T19:38:12.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.579+0000 7f078f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:12.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.579+0000 7f078f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:12.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.579+0000 7f078f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:12.579 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.579+0000 7f078f7fe640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:38:12.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.579+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0780003520 con 0x7f07900a9470 2024-03-25T19:38:12.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.581+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645468897 0 0) 0x7f07900b2400 con 0x7f07900b10d0 2024-03-25T19:38:12.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.581+0000 7f078f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:12.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.581+0000 7f078f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:12.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.581+0000 7f078f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:12.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.581+0000 7f078f7fe640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:38:12.581 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.581+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07900a9ab0 con 0x7f07900b10d0 2024-03-25T19:38:12.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.586+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311540759 0 0) 0x7f07900aa2a0 con 0x7f07900b4360 2024-03-25T19:38:12.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.586+0000 7f078f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:12.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.586+0000 7f078f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:12.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.586+0000 7f078f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:12.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.586+0000 7f078f7fe640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:38:12.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.586+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07900b2400 con 0x7f07900b4360 2024-03-25T19:38:12.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2827055789 0 0) 0x7f07900a9ab0 con 0x7f07900b10d0 2024-03-25T19:38:12.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 >> v1:172.21.15.143:6790/0 conn(0x7f07900b4360 legacy=0x7f07900b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:12.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 >> v1:172.21.15.143:6789/0 conn(0x7f07900a9470 legacy=0x7f079014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:12.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:12.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:38:12.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:12.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07900b14b0 con 0x7f07900b10d0 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0784001670 con 0x7f07900b10d0 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f078f7fe640 10 monclient: dump: 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.596 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.597 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:12.597 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:12.597 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:12.597 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:12.597 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:12.597 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:12.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f0796cb3640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:38:12.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.595+0000 7f0796cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.605+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07840017d0 con 0x7f07900b10d0 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.605+0000 7f078f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.605+0000 7f078f7fe640 1 -- 172.21.15.143:0/3015107699 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0784001be0 con 0x7f07900b10d0 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.605+0000 7f0796cb3640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.605+0000 7f0796cb3640 4 set_mon_vals no callback set 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.605+0000 7f078f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.605+0000 7f078f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.605+0000 7f078f7fe640 10 monclient: dump: 2024-03-25T19:38:12.605 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:12.606 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.606+0000 7f0796cb3640 10 monclient: shutdown 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.606+0000 7f0796cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.606+0000 7f0796cb3640 1 -- 172.21.15.143:0/3015107699 >> v1:172.21.15.203:6789/0 conn(0x7f07900b10d0 legacy=0x7f07900b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.606+0000 7f0796cb3640 1 -- 172.21.15.143:0/3015107699 shutdown_connections 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.606+0000 7f0796cb3640 1 -- 172.21.15.143:0/3015107699 >> 172.21.15.143:0/3015107699 conn(0x7f07900617a0 msgr2=0x7f0790061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.606+0000 7f0796cb3640 1 -- 172.21.15.143:0/3015107699 shutdown_connections 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.606+0000 7f0796cb3640 1 -- 172.21.15.143:0/3015107699 wait complete. 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.607+0000 7f0796cb3640 10 monclient: build_initial_monmap 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.607+0000 7f0796cb3640 10 monclient: monmap: 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:12.608 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.607+0000 7f0796cb3640 1 librados: starting msgr at 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.607+0000 7f0796cb3640 1 librados: starting objecter 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 1 Processor -- start 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 1 -- start start 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 1 librados: setting wanted keys 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 1 librados: calling monclient init 2024-03-25T19:38:12.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 10 monclient: init 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f07900a9470 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f07900b10d0 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.608+0000 7f0796cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f07900b4360 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.609+0000 7f0796cb3640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07901db420 con 0x7f07900b4360 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.609+0000 7f0796cb3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07901db8b0 con 0x7f07900b10d0 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.609+0000 7f0796cb3640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07900afe90 con 0x7f07900a9470 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.609+0000 7f0796cb3640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:12.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.609+0000 7f0796cb3640 10 monclient(hunting): authenticate will time out at 6508.952790s 2024-03-25T19:38:12.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.615+0000 7f078ffff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f07900b10d0 0x7f0790139e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:50246/0 (socket says 172.21.15.143:50246) 2024-03-25T19:38:12.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.615+0000 7f078ffff640 1 -- 172.21.15.143:0/94407074 learned_addr learned my addr 172.21.15.143:0/94407074 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:12.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.617+0000 7f078d7fa640 10 client.?.objecter ms_handle_connect 0x7f07900b10d0 2024-03-25T19:38:12.616 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.617+0000 7f078d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:12.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.618+0000 7f078d7fa640 10 client.?.objecter ms_handle_connect 0x7f07900a9470 2024-03-25T19:38:12.617 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.618+0000 7f078d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:12.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.634+0000 7f078d7fa640 10 client.?.objecter ms_handle_connect 0x7f07900b4360 2024-03-25T19:38:12.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.634+0000 7f078d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:12.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.688+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160687128 0 0) 0x7f07901db420 con 0x7f07900b4360 2024-03-25T19:38:12.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.688+0000 7f078d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:12.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.688+0000 7f078d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:12.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.688+0000 7f078d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:12.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.688+0000 7f078d7fa640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:38:12.688 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.688+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0774004eb0 con 0x7f07900b4360 2024-03-25T19:38:12.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.691+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814217418 0 0) 0x7f07901db8b0 con 0x7f07900b10d0 2024-03-25T19:38:12.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.691+0000 7f078d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:12.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.691+0000 7f078d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:12.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.691+0000 7f078d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:12.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.691+0000 7f078d7fa640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:38:12.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.691+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07901db420 con 0x7f07900b10d0 2024-03-25T19:38:12.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.695+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409160537 0 0) 0x7f07900afe90 con 0x7f07900a9470 2024-03-25T19:38:12.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.695+0000 7f078d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:12.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.695+0000 7f078d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:12.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.695+0000 7f078d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:12.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.695+0000 7f078d7fa640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:38:12.695 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.695+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07901db8b0 con 0x7f07900a9470 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.705+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284157539 0 0) 0x7f07901db8b0 con 0x7f07900a9470 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.705+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07900afe90 con 0x7f07900a9470 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.705+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0784001ae0 con 0x7f07900a9470 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.705+0000 7f078d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.705+0000 7f078d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.705+0000 7f078d7fa640 10 monclient(hunting): dump: 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:12.705 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.706 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.706 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:12.706 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:12.706 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:12.706 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:12.706 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:12.706 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:12.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.705+0000 7f078d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.713+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4259634771 0 0) 0x7f0774004eb0 con 0x7f07900b4360 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.713+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07901db8b0 con 0x7f07900b4360 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.713+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0788001af0 con 0x7f07900b4360 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.713+0000 7f078d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.713+0000 7f078d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.713+0000 7f078d7fa640 10 monclient(hunting): dump: 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:12.713 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.714 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.714 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:12.714 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:12.714 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:12.714 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:12.714 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:12.714 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:12.715 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.716+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3233508867 0 0) 0x7f07901db420 con 0x7f07900b10d0 2024-03-25T19:38:12.715 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.716+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0774004eb0 con 0x7f07900b10d0 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.716+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f077c0019b0 con 0x7f07900b10d0 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.716+0000 7f078d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.716+0000 7f078d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.716+0000 7f078d7fa640 10 monclient(hunting): dump: 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:12.716 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:12.717 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.725+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2283575829 0 0) 0x7f07900afe90 con 0x7f07900a9470 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.725+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 >> v1:172.21.15.203:6789/0 conn(0x7f07900b10d0 legacy=0x7f0790139e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.725+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 >> v1:172.21.15.143:6789/0 conn(0x7f07900b4360 legacy=0x7f07901e2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.725+0000 7f078d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.725+0000 7f078d7fa640 1 monclient: found mon.c 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f078d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f078d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07900b0020 con 0x7f07900a9470 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f078d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f078d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f078d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 10 monclient: _renew_subs 2024-03-25T19:38:12.726 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07900abfb0 con 0x7f07900a9470 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 10 monclient: _renew_subs 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f079013a780 con 0x7f07900a9470 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.726+0000 7f0796cb3640 1 librados: init done 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.727+0000 7f0796cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.727+0000 7f0796cb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:12.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.727+0000 7f0796cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:12.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:12.727+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07900b68c0 con 0x7f07900a9470 2024-03-25T19:38:13.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.025+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0784001210 con 0x7f07900a9470 2024-03-25T19:38:13.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.025+0000 7f078d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:13.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.025+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0784003a40 con 0x7f07900a9470 2024-03-25T19:38:13.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.025+0000 7f078d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:13.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.025+0000 7f078d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:13.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.025+0000 7f078d7fa640 10 monclient: dump: 2024-03-25T19:38:13.025 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:13.025 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:13.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.025+0000 7f078effd640 4 set_mon_vals no callback set 2024-03-25T19:38:13.027 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.026+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f0784019a10 con 0x7f07900a9470 2024-03-25T19:38:13.027 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.027+0000 7f078ffff640 10 monclient: get_auth_request con 0x7f077406e060 auth_method 0 2024-03-25T19:38:13.027 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.027+0000 7f078d7fa640 10 client.4586.objecter ms_handle_connect 0x7f077406e060 2024-03-25T19:38:13.032 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.032+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f0784053610 con 0x7f07900a9470 2024-03-25T19:38:13.032 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.032+0000 7f078d7fa640 10 client.4586.objecter ms_dispatch 0x7f0790000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:13.032 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.032+0000 7f078d7fa640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:13.032 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.032+0000 7f078d7fa640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:13.032 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.032+0000 7f078d7fa640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T19:38:13.032 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.032+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f078404ef50 con 0x7f07900a9470 2024-03-25T19:38:13.032 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.032+0000 7f078d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:13.032 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.032+0000 7f078d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:13.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.281+0000 7f0796cb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:13.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.281+0000 7f0796cb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:13.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.281+0000 7f0796cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:13.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.281+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07900627b0 con 0x7f07900a9470 2024-03-25T19:38:13.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.309+0000 7f078d7fa640 1 -- 172.21.15.143:0/94407074 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f0784051050 con 0x7f07900a9470 2024-03-25T19:38:13.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.310+0000 7f078d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:13.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.310+0000 7f078d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:13.310 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:38:13.310 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.312+0000 7f0796cb3640 10 librados: watch_flush enter 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.312+0000 7f0796cb3640 10 librados: watch_flush exit 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.312+0000 7f0796cb3640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.312+0000 7f0796cb3640 10 client.4586.objecter successfully canceled tick 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 >> v1:172.21.15.143:6800/1292282666 conn(0x7f077406e060 legacy=0x7f0774070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 10 monclient: shutdown 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 >> v1:172.21.15.143:6790/0 conn(0x7f07900a9470 legacy=0x7f07900ab8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 shutdown_connections 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 >> 172.21.15.143:0/94407074 conn(0x7f07900617a0 msgr2=0x7f0790061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:13.313 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 shutdown_connections 2024-03-25T19:38:13.314 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 1 -- 172.21.15.143:0/94407074 wait complete. 2024-03-25T19:38:13.314 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:13.313+0000 7f0796cb3640 1 librados: shutdown 2024-03-25T19:38:13.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:19.367 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:45.931 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:19.376 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:45.941 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:19.407 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:08:45.972 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:19.437 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:45.972 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:19.443 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:45.978 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:19.447 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:46.012 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:19.452 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:46.017 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:19.488 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:08:46.023 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:19.492 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:08:46.057 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:20.132 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:20.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.247+0000 7ff826ce2640 10 monclient: get_monmap_and_config 2024-03-25T19:38:20.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.247+0000 7ff826ce2640 10 monclient: build_initial_monmap 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.247+0000 7ff826ce2640 10 monclient: monmap: 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:38:20.248132+0000 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:38:20.248132+0000 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:38:20.247 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.248+0000 7ff826ce2640 1 Processor -- start 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.248+0000 7ff826ce2640 1 -- start start 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.248+0000 7ff826ce2640 10 monclient: init 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.248+0000 7ff826ce2640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.248+0000 7ff826ce2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.248+0000 7ff826ce2640 10 monclient(hunting): picked mon.noname-b con 0x7ff820070380 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.248+0000 7ff826ce2640 10 monclient(hunting): picked mon.noname-c con 0x7ff820070760 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.248+0000 7ff826ce2640 10 monclient(hunting): picked mon.noname-a con 0x7ff8200ae470 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.249+0000 7ff826ce2640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82013dd40 con 0x7ff8200ae470 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.249+0000 7ff826ce2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff820071740 con 0x7ff820070760 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.249+0000 7ff826ce2640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8200725d0 con 0x7ff820070380 2024-03-25T19:38:20.249 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.249+0000 7ff826ce2640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:20.250 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.249+0000 7ff826ce2640 10 monclient(hunting): authenticate will time out at 6516.592767s 2024-03-25T19:38:20.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.254+0000 7ff824a7e640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7ff820070380 0x7ff820140810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:42140/0 (socket says 172.21.15.143:42140) 2024-03-25T19:38:20.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.254+0000 7ff824a7e640 1 -- 172.21.15.143:0/791417889 learned_addr learned my addr 172.21.15.143:0/791417889 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:20.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.265+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157588766 0 0) 0x7ff8200725d0 con 0x7ff820070380 2024-03-25T19:38:20.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.265+0000 7ff8177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:20.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.265+0000 7ff8177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:20.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.265+0000 7ff8177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:20.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.265+0000 7ff8177fe640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:38:20.265 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.265+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff808003520 con 0x7ff820070380 2024-03-25T19:38:20.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.273+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084554438 0 0) 0x7ff82013dd40 con 0x7ff8200ae470 2024-03-25T19:38:20.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.273+0000 7ff8177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:20.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.273+0000 7ff8177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:20.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.273+0000 7ff8177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:20.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.273+0000 7ff8177fe640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:38:20.273 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.273+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8200725d0 con 0x7ff8200ae470 2024-03-25T19:38:20.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2429367331 0 0) 0x7ff808003520 con 0x7ff820070380 2024-03-25T19:38:20.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 >> v1:172.21.15.203:6789/0 conn(0x7ff820070760 legacy=0x7ff82013cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:20.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 >> v1:172.21.15.143:6789/0 conn(0x7ff8200ae470 legacy=0x7ff82013d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:20.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:20.277 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 1 monclient: found mon.noname-b 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8200728b0 con 0x7ff820070380 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff81c001690 con 0x7ff820070380 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff8177fe640 10 monclient: dump: 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:20.278 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff826ce2640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:38:20.279 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.277+0000 7ff826ce2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.290+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff81c0017f0 con 0x7ff820070380 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.290+0000 7ff8177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.290+0000 7ff8177fe640 1 -- 172.21.15.143:0/791417889 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff81c001c00 con 0x7ff820070380 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.290+0000 7ff8177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.290+0000 7ff8177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.290+0000 7ff8177fe640 10 monclient: dump: 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:20.290 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.290+0000 7ff826ce2640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:20.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.290+0000 7ff826ce2640 4 set_mon_vals no callback set 2024-03-25T19:38:20.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.291+0000 7ff826ce2640 10 monclient: shutdown 2024-03-25T19:38:20.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.291+0000 7ff826ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:20.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.291+0000 7ff826ce2640 1 -- 172.21.15.143:0/791417889 >> v1:172.21.15.143:6790/0 conn(0x7ff820070380 legacy=0x7ff820140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:20.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.291+0000 7ff826ce2640 1 -- 172.21.15.143:0/791417889 shutdown_connections 2024-03-25T19:38:20.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.291+0000 7ff826ce2640 1 -- 172.21.15.143:0/791417889 >> 172.21.15.143:0/791417889 conn(0x7ff82006b630 msgr2=0x7ff8200b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:20.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.291+0000 7ff826ce2640 1 -- 172.21.15.143:0/791417889 shutdown_connections 2024-03-25T19:38:20.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.291+0000 7ff826ce2640 1 -- 172.21.15.143:0/791417889 wait complete. 2024-03-25T19:38:20.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 10 monclient: build_initial_monmap 2024-03-25T19:38:20.294 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 10 monclient: monmap: 2024-03-25T19:38:20.294 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:20.294 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:20.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 1 librados: starting msgr at 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 1 librados: starting objecter 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 1 Processor -- start 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 1 -- start start 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 1 librados: setting wanted keys 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 1 librados: calling monclient init 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.292+0000 7ff826ce2640 10 monclient: init 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 10 monclient(hunting): picked mon.noname-a con 0x7ff820070380 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 10 monclient(hunting): picked mon.noname-b con 0x7ff820070760 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 10 monclient(hunting): picked mon.noname-c con 0x7ff8200ae470 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff820143cc0 con 0x7ff820070380 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff820135c90 con 0x7ff820070760 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff820141130 con 0x7ff8200ae470 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:20.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.293+0000 7ff826ce2640 10 monclient(hunting): authenticate will time out at 6516.637245s 2024-03-25T19:38:20.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.308+0000 7ff82527f640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7ff8200ae470 0x7ff8201e2240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:42170/0 (socket says 172.21.15.143:42170) 2024-03-25T19:38:20.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.308+0000 7ff82527f640 1 -- 172.21.15.143:0/543641333 learned_addr learned my addr 172.21.15.143:0/543641333 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:20.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.309+0000 7ff8157fa640 10 client.?.objecter ms_handle_connect 0x7ff8200ae470 2024-03-25T19:38:20.309 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.309+0000 7ff8157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:20.315 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.316+0000 7ff8157fa640 10 client.?.objecter ms_handle_connect 0x7ff820070760 2024-03-25T19:38:20.315 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.316+0000 7ff8157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:20.317 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.317+0000 7ff8157fa640 10 client.?.objecter ms_handle_connect 0x7ff820070380 2024-03-25T19:38:20.317 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.317+0000 7ff8157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:20.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.330+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367370635 0 0) 0x7ff820135c90 con 0x7ff820070760 2024-03-25T19:38:20.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.330+0000 7ff8157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:20.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.330+0000 7ff8157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:20.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.330+0000 7ff8157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:20.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.330+0000 7ff8157fa640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:38:20.330 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.330+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80c004eb0 con 0x7ff820070760 2024-03-25T19:38:20.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.331+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023066120 0 0) 0x7ff820143cc0 con 0x7ff820070380 2024-03-25T19:38:20.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.331+0000 7ff8157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:20.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.331+0000 7ff8157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:20.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.331+0000 7ff8157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:20.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.331+0000 7ff8157fa640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:38:20.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.331+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff820135c90 con 0x7ff820070380 2024-03-25T19:38:20.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.332+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079998517 0 0) 0x7ff820141130 con 0x7ff8200ae470 2024-03-25T19:38:20.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.332+0000 7ff8157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:20.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.332+0000 7ff8157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:20.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.332+0000 7ff8157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:20.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.332+0000 7ff8157fa640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:38:20.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.332+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff820143cc0 con 0x7ff8200ae470 2024-03-25T19:38:20.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.339+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 635333798 0 0) 0x7ff820143cc0 con 0x7ff8200ae470 2024-03-25T19:38:20.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.339+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff820141130 con 0x7ff8200ae470 2024-03-25T19:38:20.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.339+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff818001bd0 con 0x7ff8200ae470 2024-03-25T19:38:20.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.339+0000 7ff8157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:20.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.339+0000 7ff8157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:20.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.339+0000 7ff8157fa640 10 monclient(hunting): dump: 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:20.340 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.339+0000 7ff8157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:20.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.346+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2214161798 0 0) 0x7ff80c004eb0 con 0x7ff820070760 2024-03-25T19:38:20.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.346+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff820143cc0 con 0x7ff820070760 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.346+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff810001930 con 0x7ff820070760 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.346+0000 7ff8157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.346+0000 7ff8157fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.346+0000 7ff8157fa640 10 monclient(hunting): dump: 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:20.346 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:20.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2490267931 0 0) 0x7ff820141130 con 0x7ff8200ae470 2024-03-25T19:38:20.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 >> v1:172.21.15.203:6789/0 conn(0x7ff820070760 legacy=0x7ff82006dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:20.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 >> v1:172.21.15.143:6789/0 conn(0x7ff820070380 legacy=0x7ff8201422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:20.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 10 monclient: _finish_hunting 0 2024-03-25T19:38:20.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 1 monclient: found mon.c 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8201412c0 con 0x7ff8200ae470 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 10 monclient: _finish_auth 0 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 10 monclient: _check_auth_tickets 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff8157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 5 monclient: authenticate success, global_id 4592 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 10 monclient: _renew_subs 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff820142ca0 con 0x7ff8200ae470 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 10 monclient: _renew_subs 2024-03-25T19:38:20.349 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:20.350 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff820141b20 con 0x7ff8200ae470 2024-03-25T19:38:20.350 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.348+0000 7ff826ce2640 1 librados: init done 2024-03-25T19:38:20.350 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.349+0000 7ff826ce2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:20.350 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.349+0000 7ff826ce2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:20.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.349+0000 7ff826ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:20.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.349+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff820071b00 con 0x7ff8200ae470 2024-03-25T19:38:20.632 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.632+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8180020a0 con 0x7ff8200ae470 2024-03-25T19:38:20.632 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.632+0000 7ff8157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:20.632 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.632+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff818003b30 con 0x7ff8200ae470 2024-03-25T19:38:20.632 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.632+0000 7ff8157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.632+0000 7ff8157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.632+0000 7ff8157fa640 10 monclient: dump: 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:20.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.632+0000 7ff816ffd640 4 set_mon_vals no callback set 2024-03-25T19:38:20.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.633+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7ff818003d50 con 0x7ff8200ae470 2024-03-25T19:38:20.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.634+0000 7ff824a7e640 10 monclient: get_auth_request con 0x7ff80c06df20 auth_method 0 2024-03-25T19:38:20.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.634+0000 7ff8157fa640 10 client.4592.objecter ms_handle_connect 0x7ff80c06df20 2024-03-25T19:38:20.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.641+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7ff8180549e0 con 0x7ff8200ae470 2024-03-25T19:38:20.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.641+0000 7ff8157fa640 10 client.4592.objecter ms_dispatch 0x7ff820000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:20.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.641+0000 7ff8157fa640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:20.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.641+0000 7ff8157fa640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:20.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.641+0000 7ff8157fa640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T19:38:20.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.641+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff818050320 con 0x7ff8200ae470 2024-03-25T19:38:20.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.641+0000 7ff8157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:20.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.641+0000 7ff8157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:20.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.889+0000 7ff826ce2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:20.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.889+0000 7ff826ce2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:20.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.889+0000 7ff826ce2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:20.889 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.889+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8201429c0 con 0x7ff8200ae470 2024-03-25T19:38:20.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.917+0000 7ff8157fa640 1 -- 172.21.15.143:0/543641333 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7ff8201429c0 con 0x7ff8200ae470 2024-03-25T19:38:20.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.917+0000 7ff8157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:20.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.917+0000 7ff8157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:20.917 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:38:20.918 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:20.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 10 librados: watch_flush enter 2024-03-25T19:38:20.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 10 librados: watch_flush exit 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 10 client.4592.objecter successfully canceled tick 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 >> v1:172.21.15.143:6800/1292282666 conn(0x7ff80c06df20 legacy=0x7ff80c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 10 monclient: shutdown 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 >> v1:172.21.15.143:6790/0 conn(0x7ff8200ae470 legacy=0x7ff8201e2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 shutdown_connections 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.920+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 >> 172.21.15.143:0/543641333 conn(0x7ff82006b630 msgr2=0x7ff8200b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.921+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 shutdown_connections 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.921+0000 7ff826ce2640 1 -- 172.21.15.143:0/543641333 wait complete. 2024-03-25T19:38:20.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:20.921+0000 7ff826ce2640 1 librados: shutdown 2024-03-25T19:38:20.932 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:27.738 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:27.852 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.853+0000 7fd988c17640 10 monclient: get_monmap_and_config 2024-03-25T19:38:27.852 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.853+0000 7fd988c17640 10 monclient: build_initial_monmap 2024-03-25T19:38:27.852 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.853+0000 7fd988c17640 10 monclient: monmap: 2024-03-25T19:38:27.852 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:38:27.854010+0000 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:38:27.854010+0000 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:38:27.853 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:27.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 1 Processor -- start 2024-03-25T19:38:27.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 1 -- start start 2024-03-25T19:38:27.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 10 monclient: init 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 10 monclient(hunting): picked mon.noname-c con 0x7fd9840b4aa0 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 10 monclient(hunting): picked mon.noname-b con 0x7fd9840b4e80 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 10 monclient(hunting): picked mon.noname-a con 0x7fd9840b2140 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9840b0fe0 con 0x7fd9840b2140 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98414ae50 con 0x7fd9840b4aa0 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9840b1230 con 0x7fd9840b4e80 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:27.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.854+0000 7fd988c17640 10 monclient(hunting): authenticate will time out at 6524.198563s 2024-03-25T19:38:27.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.859+0000 7fd98259c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd9840b4aa0 0x7fd98414a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:45342/0 (socket says 172.21.15.143:45342) 2024-03-25T19:38:27.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.859+0000 7fd98259c640 1 -- 172.21.15.143:0/1197051763 learned_addr learned my addr 172.21.15.143:0/1197051763 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:27.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.871+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618100276 0 0) 0x7fd9840b0fe0 con 0x7fd9840b2140 2024-03-25T19:38:27.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:27.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:27.871 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:27.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:38:27.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd970003520 con 0x7fd9840b2140 2024-03-25T19:38:27.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868706334 0 0) 0x7fd98414ae50 con 0x7fd9840b4aa0 2024-03-25T19:38:27.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:27.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:27.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:27.872 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:38:27.873 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.872+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9840b0fe0 con 0x7fd9840b4aa0 2024-03-25T19:38:27.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.875+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478438943 0 0) 0x7fd9840b1230 con 0x7fd9840b4e80 2024-03-25T19:38:27.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.875+0000 7fd981d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:38:27.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.875+0000 7fd981d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:27.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.875+0000 7fd981d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:27.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.875+0000 7fd981d9b640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:38:27.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.875+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd98414ae50 con 0x7fd9840b4e80 2024-03-25T19:38:27.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 256627960 0 0) 0x7fd970003520 con 0x7fd9840b2140 2024-03-25T19:38:27.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 >> v1:172.21.15.143:6790/0 conn(0x7fd9840b4e80 legacy=0x7fd9840b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:27.881 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 >> v1:172.21.15.203:6789/0 conn(0x7fd9840b4aa0 legacy=0x7fd98414a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 1 monclient: found mon.noname-a 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9840b0000 con 0x7fd9840b2140 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 10 monclient: _finish_auth 0 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd97c001a10 con 0x7fd9840b2140 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd981d9b640 10 monclient: dump: 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:27.882 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd988c17640 5 monclient: authenticate success, global_id 4605 2024-03-25T19:38:27.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.881+0000 7fd988c17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:27.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.892+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd97c001b70 con 0x7fd9840b2140 2024-03-25T19:38:27.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.892+0000 7fd981d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:27.891 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.892+0000 7fd981d9b640 1 -- 172.21.15.143:0/1197051763 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd97c001f80 con 0x7fd9840b2140 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.892+0000 7fd981d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.892+0000 7fd981d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.892+0000 7fd981d9b640 10 monclient: dump: 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:27.892 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.892+0000 7fd988c17640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:27.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.892+0000 7fd988c17640 4 set_mon_vals no callback set 2024-03-25T19:38:27.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.893+0000 7fd988c17640 10 monclient: shutdown 2024-03-25T19:38:27.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.893+0000 7fd988c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:27.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.893+0000 7fd988c17640 1 -- 172.21.15.143:0/1197051763 >> v1:172.21.15.143:6789/0 conn(0x7fd9840b2140 legacy=0x7fd9841522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:27.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.893+0000 7fd988c17640 1 -- 172.21.15.143:0/1197051763 shutdown_connections 2024-03-25T19:38:27.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.893+0000 7fd988c17640 1 -- 172.21.15.143:0/1197051763 >> 172.21.15.143:0/1197051763 conn(0x7fd9840617a0 msgr2=0x7fd984061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:27.894 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.893+0000 7fd988c17640 1 -- 172.21.15.143:0/1197051763 shutdown_connections 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.893+0000 7fd988c17640 1 -- 172.21.15.143:0/1197051763 wait complete. 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 10 monclient: build_initial_monmap 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 10 monclient: monmap: 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 1 librados: starting msgr at 2024-03-25T19:38:27.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 1 librados: starting objecter 2024-03-25T19:38:27.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 1 Processor -- start 2024-03-25T19:38:27.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 1 -- start start 2024-03-25T19:38:27.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 1 librados: setting wanted keys 2024-03-25T19:38:27.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 1 librados: calling monclient init 2024-03-25T19:38:27.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.894+0000 7fd988c17640 10 monclient: init 2024-03-25T19:38:27.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:27.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:38:27.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 10 monclient(hunting): picked mon.noname-c con 0x7fd9840b2140 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:27.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 10 monclient(hunting): picked mon.noname-a con 0x7fd9840b4aa0 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:27.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 10 monclient(hunting): picked mon.noname-b con 0x7fd984070ef0 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:27.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98414a0d0 con 0x7fd9840b4aa0 2024-03-25T19:38:27.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd984071d50 con 0x7fd984070ef0 2024-03-25T19:38:27.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9841dfa00 con 0x7fd9840b2140 2024-03-25T19:38:27.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:27.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.895+0000 7fd988c17640 10 monclient(hunting): authenticate will time out at 6524.239469s 2024-03-25T19:38:27.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.912+0000 7fd98259c640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fd9840b2140 0x7fd9841dff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:46034/0 (socket says 172.21.15.143:46034) 2024-03-25T19:38:27.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.912+0000 7fd98259c640 1 -- 172.21.15.143:0/1190858952 learned_addr learned my addr 172.21.15.143:0/1190858952 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:27.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.915+0000 7fd97affd640 10 client.?.objecter ms_handle_connect 0x7fd9840b2140 2024-03-25T19:38:27.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.915+0000 7fd97affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:27.921 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.922+0000 7fd97affd640 10 client.?.objecter ms_handle_connect 0x7fd984070ef0 2024-03-25T19:38:27.922 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.922+0000 7fd97affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:27.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.928+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5305195 0 0) 0x7fd984071d50 con 0x7fd984070ef0 2024-03-25T19:38:27.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.928+0000 7fd97affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:27.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.928+0000 7fd97affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:27.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.928+0000 7fd97affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:27.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.928+0000 7fd97affd640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:38:27.928 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.928+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd964004eb0 con 0x7fd984070ef0 2024-03-25T19:38:27.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.931+0000 7fd97affd640 10 client.?.objecter ms_handle_connect 0x7fd9840b4aa0 2024-03-25T19:38:27.931 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.931+0000 7fd97affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:27.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.936+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895000182 0 0) 0x7fd9841dfa00 con 0x7fd9840b2140 2024-03-25T19:38:27.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.936+0000 7fd97affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:27.935 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.936+0000 7fd97affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:27.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.936+0000 7fd97affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:27.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.936+0000 7fd97affd640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:38:27.936 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.936+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd984071d50 con 0x7fd9840b2140 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.942+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1012619627 0 0) 0x7fd964004eb0 con 0x7fd984070ef0 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.942+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9841dfa00 con 0x7fd984070ef0 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.942+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd97c001e80 con 0x7fd984070ef0 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.942+0000 7fd97affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.942+0000 7fd97affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.942+0000 7fd97affd640 10 monclient(hunting): dump: 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:27.942 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:27.943 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:27.943 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:27.943 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:27.943 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:27.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.942+0000 7fd97affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:27.953 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.951+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64074478 0 0) 0x7fd98414a0d0 con 0x7fd9840b4aa0 2024-03-25T19:38:27.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.951+0000 7fd97affd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:27.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.951+0000 7fd97affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:27.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.951+0000 7fd97affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:27.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.951+0000 7fd97affd640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:38:27.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.951+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd964004eb0 con 0x7fd9840b4aa0 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.959+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1005842667 0 0) 0x7fd9841dfa00 con 0x7fd984070ef0 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 >> v1:172.21.15.143:6790/0 conn(0x7fd9840b2140 legacy=0x7fd9841dff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 >> v1:172.21.15.143:6789/0 conn(0x7fd9840b4aa0 legacy=0x7fd98413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 1 monclient: found mon.b 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9841df690 con 0x7fd984070ef0 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 10 monclient: _finish_auth 0 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd97affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:27.960 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 5 monclient: authenticate success, global_id 4612 2024-03-25T19:38:27.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 10 monclient: _renew_subs 2024-03-25T19:38:27.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:27.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd984147480 con 0x7fd984070ef0 2024-03-25T19:38:27.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:27.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 10 monclient: _renew_subs 2024-03-25T19:38:27.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:27.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9840ae260 con 0x7fd984070ef0 2024-03-25T19:38:27.961 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.960+0000 7fd988c17640 1 librados: init done 2024-03-25T19:38:27.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.961+0000 7fd988c17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:27.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.961+0000 7fd988c17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:27.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.961+0000 7fd988c17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:27.962 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:27.961+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9841dfb90 con 0x7fd984070ef0 2024-03-25T19:38:28.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.237+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd97c002040 con 0x7fd984070ef0 2024-03-25T19:38:28.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.237+0000 7fd97affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.237+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fd97c003dc0 con 0x7fd984070ef0 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.237+0000 7fd97affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.237+0000 7fd97affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.237+0000 7fd97affd640 10 monclient: dump: 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:28.238 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:28.239 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:28.239 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:28.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.237+0000 7fd98159a640 4 set_mon_vals no callback set 2024-03-25T19:38:28.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.238+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fd97c019d90 con 0x7fd984070ef0 2024-03-25T19:38:28.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.239+0000 7fd98259c640 10 monclient: get_auth_request con 0x7fd96406de00 auth_method 0 2024-03-25T19:38:28.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.239+0000 7fd97affd640 10 client.4612.objecter ms_handle_connect 0x7fd96406de00 2024-03-25T19:38:28.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.245+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fd97c053810 con 0x7fd984070ef0 2024-03-25T19:38:28.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.245+0000 7fd97affd640 10 client.4612.objecter ms_dispatch 0x7fd984000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:28.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.245+0000 7fd97affd640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:28.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.245+0000 7fd97affd640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:28.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.245+0000 7fd97affd640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T19:38:28.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.248+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd97c04f150 con 0x7fd984070ef0 2024-03-25T19:38:28.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.248+0000 7fd97affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:28.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.248+0000 7fd97affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:28.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.489+0000 7fd988c17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:28.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.489+0000 7fd988c17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:28.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.489+0000 7fd988c17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:28.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.489+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd984147610 con 0x7fd984070ef0 2024-03-25T19:38:28.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.530+0000 7fd97affd640 1 -- 172.21.15.143:0/1190858952 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fd97c051250 con 0x7fd984070ef0 2024-03-25T19:38:28.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.530+0000 7fd97affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:28.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.530+0000 7fd97affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:28.530 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:38:28.531 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:28.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 10 librados: watch_flush enter 2024-03-25T19:38:28.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 10 librados: watch_flush exit 2024-03-25T19:38:28.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T19:38:28.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 10 client.4612.objecter successfully canceled tick 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 >> v1:172.21.15.143:6800/1292282666 conn(0x7fd96406de00 legacy=0x7fd9640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 10 monclient: shutdown 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 >> v1:172.21.15.203:6789/0 conn(0x7fd984070ef0 legacy=0x7fd9841e79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 shutdown_connections 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 >> 172.21.15.143:0/1190858952 conn(0x7fd9840617a0 msgr2=0x7fd984061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 shutdown_connections 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 1 -- 172.21.15.143:0/1190858952 wait complete. 2024-03-25T19:38:28.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:28.533+0000 7fd988c17640 1 librados: shutdown 2024-03-25T19:38:28.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:35.351 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:35.464 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.465+0000 7f213c833640 10 monclient: get_monmap_and_config 2024-03-25T19:38:35.464 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.465+0000 7f213c833640 10 monclient: build_initial_monmap 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.465+0000 7f213c833640 10 monclient: monmap: 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:38:35.466059+0000 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:38:35.466059+0000 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:38:35.465 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:35.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.465+0000 7f213c833640 1 Processor -- start 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.465+0000 7f213c833640 1 -- start start 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.465+0000 7f213c833640 10 monclient: init 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 10 monclient(hunting): picked mon.noname-c con 0x7f21380b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 10 monclient(hunting): picked mon.noname-a con 0x7f21380b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 10 monclient(hunting): picked mon.noname-b con 0x7f213814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380af940 con 0x7f21380b1730 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380b0a70 con 0x7f21380b4d60 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21380b0d50 con 0x7f213814be90 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:35.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.466+0000 7f213c833640 10 monclient(hunting): authenticate will time out at 6531.810614s 2024-03-25T19:38:35.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.480+0000 7f2136d9d640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f213814be90 0x7f213814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:50872/0 (socket says 172.21.15.143:50872) 2024-03-25T19:38:35.480 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.480+0000 7f2136d9d640 1 -- 172.21.15.143:0/2843498783 learned_addr learned my addr 172.21.15.143:0/2843498783 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:35.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.486+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176155338 0 0) 0x7f21380b0a70 con 0x7f21380b4d60 2024-03-25T19:38:35.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.486+0000 7f213559a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:35.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.486+0000 7f213559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:35.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.486+0000 7f213559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:35.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.486+0000 7f213559a640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:38:35.486 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.486+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2120003520 con 0x7f21380b4d60 2024-03-25T19:38:35.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.495+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064638826 0 0) 0x7f21380b0d50 con 0x7f213814be90 2024-03-25T19:38:35.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.495+0000 7f213559a640 10 monclient(hunting): _negotiate 2024-03-25T19:38:35.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.495+0000 7f213559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:35.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.495+0000 7f213559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:35.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.495+0000 7f213559a640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:38:35.495 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.495+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21380b0a70 con 0x7f213814be90 2024-03-25T19:38:35.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3814984284 0 0) 0x7f2120003520 con 0x7f21380b4d60 2024-03-25T19:38:35.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 >> v1:172.21.15.143:6790/0 conn(0x7f213814be90 legacy=0x7f213814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:35.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 >> v1:172.21.15.143:6789/0 conn(0x7f21380b1730 legacy=0x7f21380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:35.503 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 10 monclient: _finish_hunting 0 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 1 monclient: found mon.noname-c 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21380b1030 con 0x7f21380b4d60 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 10 monclient: _finish_auth 0 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 10 monclient: _check_auth_tickets 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2128001690 con 0x7f21380b4d60 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213559a640 10 monclient: dump: 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:35.504 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:35.505 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:35.505 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:35.505 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:35.505 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:35.505 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213c833640 5 monclient: authenticate success, global_id 4615 2024-03-25T19:38:35.505 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.503+0000 7f213c833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.512+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2128001870 con 0x7f21380b4d60 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.512+0000 7f213559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.512+0000 7f213559a640 1 -- 172.21.15.143:0/2843498783 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2128001c80 con 0x7f21380b4d60 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.512+0000 7f213559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.512+0000 7f213559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.512+0000 7f213559a640 10 monclient: dump: 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:35.512 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:35.513 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:35.513 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:35.513 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:35.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.512+0000 7f213c833640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:35.513 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.512+0000 7f213c833640 4 set_mon_vals no callback set 2024-03-25T19:38:35.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.513+0000 7f213c833640 10 monclient: shutdown 2024-03-25T19:38:35.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.513+0000 7f213c833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:35.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.513+0000 7f213c833640 1 -- 172.21.15.143:0/2843498783 >> v1:172.21.15.203:6789/0 conn(0x7f21380b4d60 legacy=0x7f213813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:35.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.513+0000 7f213c833640 1 -- 172.21.15.143:0/2843498783 shutdown_connections 2024-03-25T19:38:35.514 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.513+0000 7f213c833640 1 -- 172.21.15.143:0/2843498783 >> 172.21.15.143:0/2843498783 conn(0x7f21380617a0 msgr2=0x7f2138061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.513+0000 7f213c833640 1 -- 172.21.15.143:0/2843498783 shutdown_connections 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.513+0000 7f213c833640 1 -- 172.21.15.143:0/2843498783 wait complete. 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 10 monclient: build_initial_monmap 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 10 monclient: monmap: 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:38:35.515 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:35.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 1 librados: starting msgr at 2024-03-25T19:38:35.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 1 librados: starting objecter 2024-03-25T19:38:35.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 1 Processor -- start 2024-03-25T19:38:35.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 1 -- start start 2024-03-25T19:38:35.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 1 librados: setting wanted keys 2024-03-25T19:38:35.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 1 librados: calling monclient init 2024-03-25T19:38:35.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.514+0000 7f213c833640 10 monclient: init 2024-03-25T19:38:35.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:35.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:35.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 10 monclient(hunting): picked mon.noname-b con 0x7f21380b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:35.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 10 monclient(hunting): picked mon.noname-a con 0x7f21380b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:35.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 10 monclient(hunting): picked mon.noname-c con 0x7f213814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:35.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21381d7940 con 0x7f21380b4d60 2024-03-25T19:38:35.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138142140 con 0x7f21380b1730 2024-03-25T19:38:35.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f213813be00 con 0x7f213814be90 2024-03-25T19:38:35.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:35.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.515+0000 7f213c833640 10 monclient(hunting): authenticate will time out at 6531.859417s 2024-03-25T19:38:35.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.528+0000 7f2135d9b640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f21380b4d60 0x7f21381d7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:49786/0 (socket says 172.21.15.143:49786) 2024-03-25T19:38:35.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.528+0000 7f2135d9b640 1 -- 172.21.15.143:0/2884791326 learned_addr learned my addr 172.21.15.143:0/2884791326 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:35.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.530+0000 7f2126ffd640 10 client.?.objecter ms_handle_connect 0x7f213814be90 2024-03-25T19:38:35.530 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.530+0000 7f2126ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:35.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.533+0000 7f2126ffd640 10 client.?.objecter ms_handle_connect 0x7f21380b4d60 2024-03-25T19:38:35.533 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.533+0000 7f2126ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:35.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.534+0000 7f2126ffd640 10 client.?.objecter ms_handle_connect 0x7f21380b1730 2024-03-25T19:38:35.534 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.534+0000 7f2126ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:35.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.547+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113247696 0 0) 0x7f2138142140 con 0x7f21380b1730 2024-03-25T19:38:35.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.547+0000 7f2126ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:35.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.547+0000 7f2126ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:35.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.547+0000 7f2126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:35.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.547+0000 7f2126ffd640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:38:35.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.547+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2118004eb0 con 0x7f21380b1730 2024-03-25T19:38:35.556 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.557+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643850707 0 0) 0x7f21381d7940 con 0x7f21380b4d60 2024-03-25T19:38:35.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.557+0000 7f2126ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:35.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.557+0000 7f2126ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:35.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.557+0000 7f2126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:35.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.557+0000 7f2126ffd640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:38:35.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.557+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2138142140 con 0x7f21380b4d60 2024-03-25T19:38:35.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.558+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131193081 0 0) 0x7f213813be00 con 0x7f213814be90 2024-03-25T19:38:35.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.558+0000 7f2126ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:35.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.558+0000 7f2126ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:35.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.558+0000 7f2126ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:35.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.558+0000 7f2126ffd640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:38:35.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.558+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21381d7940 con 0x7f213814be90 2024-03-25T19:38:35.573 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.574+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 676781250 0 0) 0x7f2118004eb0 con 0x7f21380b1730 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.574+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f213813be00 con 0x7f21380b1730 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.574+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2128001bc0 con 0x7f21380b1730 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.574+0000 7f2126ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.574+0000 7f2126ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.574+0000 7f2126ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:35.574 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.574+0000 7f2126ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:35.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.587+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3486064193 0 0) 0x7f2138142140 con 0x7f21380b4d60 2024-03-25T19:38:35.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.587+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2118004eb0 con 0x7f21380b4d60 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.587+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f212c001970 con 0x7f21380b4d60 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.587+0000 7f2126ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.587+0000 7f2126ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.587+0000 7f2126ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:35.587 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:35.589 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.590+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4128990884 0 0) 0x7f21381d7940 con 0x7f213814be90 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.590+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2138142140 con 0x7f213814be90 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.590+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2130001bd0 con 0x7f213814be90 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.590+0000 7f2126ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.590+0000 7f2126ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.590+0000 7f2126ffd640 10 monclient(hunting): dump: 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:35.590 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.602+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1843433522 0 0) 0x7f213813be00 con 0x7f21380b1730 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.602+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 >> v1:172.21.15.143:6790/0 conn(0x7f213814be90 legacy=0x7f2138142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.602+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 >> v1:172.21.15.143:6789/0 conn(0x7f21380b4d60 legacy=0x7f21381d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.602+0000 7f2126ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.602+0000 7f2126ffd640 1 monclient: found mon.b 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f2126ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f2126ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21381d8530 con 0x7f21380b1730 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f2126ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f2126ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f2126ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:38:35.603 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 10 monclient: _renew_subs 2024-03-25T19:38:35.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:35.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21381d8780 con 0x7f21380b1730 2024-03-25T19:38:35.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:35.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 10 monclient: _renew_subs 2024-03-25T19:38:35.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:35.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21381427b0 con 0x7f21380b1730 2024-03-25T19:38:35.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.603+0000 7f213c833640 1 librados: init done 2024-03-25T19:38:35.604 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.604+0000 7f213c833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:35.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.604+0000 7f213c833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:35.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.604+0000 7f213c833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:35.605 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.604+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21381401f0 con 0x7f21380b1730 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.902+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2128003960 con 0x7f21380b1730 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.902+0000 7f2126ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.902+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f2128001250 con 0x7f21380b1730 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.902+0000 7f2126ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.902+0000 7f2126ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.902+0000 7f2126ffd640 10 monclient: dump: 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:35.902 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:35.903 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:35.903 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:35.903 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:35.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.902+0000 7f2134d99640 4 set_mon_vals no callback set 2024-03-25T19:38:35.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.903+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f2128019ae0 con 0x7f21380b1730 2024-03-25T19:38:35.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.904+0000 7f2135d9b640 10 monclient: get_auth_request con 0x7f211806e0b0 auth_method 0 2024-03-25T19:38:35.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.904+0000 7f2126ffd640 10 client.4618.objecter ms_handle_connect 0x7f211806e0b0 2024-03-25T19:38:35.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.908+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f2128053790 con 0x7f21380b1730 2024-03-25T19:38:35.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.908+0000 7f2126ffd640 10 client.4618.objecter ms_dispatch 0x7f2138000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:35.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.908+0000 7f2126ffd640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:35.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.908+0000 7f2126ffd640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:35.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.908+0000 7f2126ffd640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T19:38:35.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.909+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f212804f2a0 con 0x7f21380b1730 2024-03-25T19:38:35.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.909+0000 7f2126ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:35.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:35.909+0000 7f2126ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:36.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.152+0000 7f213c833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:36.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.152+0000 7f213c833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:36.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.152+0000 7f213c833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:38:36.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.152+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21380627b0 con 0x7f21380b1730 2024-03-25T19:38:36.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.293+0000 7f2126ffd640 1 -- 172.21.15.143:0/2884791326 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f21280511d0 con 0x7f21380b1730 2024-03-25T19:38:36.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.293+0000 7f2126ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:36.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.293+0000 7f2126ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:36.293 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:38:36.293 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.295+0000 7f213c833640 10 librados: watch_flush enter 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.295+0000 7f213c833640 10 librados: watch_flush exit 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.295+0000 7f213c833640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.295+0000 7f213c833640 10 client.4618.objecter successfully canceled tick 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 >> v1:172.21.15.143:6800/1292282666 conn(0x7f211806e0b0 legacy=0x7f2118070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 10 monclient: shutdown 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 >> v1:172.21.15.203:6789/0 conn(0x7f21380b1730 legacy=0x7f213813b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 shutdown_connections 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 >> 172.21.15.143:0/2884791326 conn(0x7f21380617a0 msgr2=0x7f213813c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 shutdown_connections 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 1 -- 172.21.15.143:0/2884791326 wait complete. 2024-03-25T19:38:36.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:36.296+0000 7f213c833640 1 librados: shutdown 2024-03-25T19:38:36.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:39.426 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:05.990 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.441 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:06.006 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.473 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:06.037 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.492 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:06.057 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.495 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:06.031 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.502 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:06.038 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.504 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:06.068 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.524 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:06.059 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:39.531 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:06.095 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:43.114 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:43.227 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.228+0000 7fb1ce58c640 10 monclient: get_monmap_and_config 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.228+0000 7fb1ce58c640 10 monclient: build_initial_monmap 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.228+0000 7fb1ce58c640 10 monclient: monmap: 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:38:43.229210+0000 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:38:43.229210+0000 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:38:43.228 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:43.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 1 Processor -- start 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 1 -- start start 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 10 monclient: init 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 10 monclient(hunting): picked mon.noname-b con 0x7fb1c80b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 10 monclient(hunting): picked mon.noname-c con 0x7fb1c80b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 10 monclient(hunting): picked mon.noname-a con 0x7fb1c814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c80af940 con 0x7fb1c814be90 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c80b0a70 con 0x7fb1c80b1730 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c80b0d50 con 0x7fb1c80b4d60 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:43.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.229+0000 7fb1ce58c640 10 monclient(hunting): authenticate will time out at 6539.573602s 2024-03-25T19:38:43.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.235+0000 7fb1c77fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb1c80b1730 0x7fb1c80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:46802/0 (socket says 172.21.15.143:46802) 2024-03-25T19:38:43.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.235+0000 7fb1c77fe640 1 -- 172.21.15.143:0/1412124058 learned_addr learned my addr 172.21.15.143:0/1412124058 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:43.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.245+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268722855 0 0) 0x7fb1c80b0a70 con 0x7fb1c80b1730 2024-03-25T19:38:43.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.245+0000 7fb1c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:43.244 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.245+0000 7fb1c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:43.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.245+0000 7fb1c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:43.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.245+0000 7fb1c6ffd640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:38:43.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.245+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b0003520 con 0x7fb1c80b1730 2024-03-25T19:38:43.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.251+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312179898 0 0) 0x7fb1c80b0d50 con 0x7fb1c80b4d60 2024-03-25T19:38:43.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.251+0000 7fb1c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:38:43.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.251+0000 7fb1c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:43.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.251+0000 7fb1c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:43.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.251+0000 7fb1c6ffd640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:38:43.251 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.251+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c80b0a70 con 0x7fb1c80b4d60 2024-03-25T19:38:43.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190447321 0 0) 0x7fb1b0003520 con 0x7fb1c80b1730 2024-03-25T19:38:43.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 >> v1:172.21.15.143:6790/0 conn(0x7fb1c80b4d60 legacy=0x7fb1c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:43.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 >> v1:172.21.15.143:6789/0 conn(0x7fb1c814be90 legacy=0x7fb1c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:43.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:38:43.253 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c80b1030 con 0x7fb1c80b1730 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb1bc001690 con 0x7fb1c80b1730 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1c6ffd640 10 monclient: dump: 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:43.254 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:43.255 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:43.255 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:43.255 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:43.255 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:43.255 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:43.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1ce58c640 5 monclient: authenticate success, global_id 4621 2024-03-25T19:38:43.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.253+0000 7fb1ce58c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:43.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.256+0000 7fb1c6ffd640 1 -- 172.21.15.143:0/1412124058 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1bc0017f0 con 0x7fb1c80b1730 2024-03-25T19:38:43.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.256+0000 7fb1c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:43.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.256+0000 7fb1ce58c640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:43.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.256+0000 7fb1ce58c640 4 set_mon_vals no callback set 2024-03-25T19:38:43.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.256+0000 7fb1ce58c640 10 monclient: shutdown 2024-03-25T19:38:43.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.257+0000 7fb1ce58c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:43.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.257+0000 7fb1ce58c640 1 -- 172.21.15.143:0/1412124058 >> v1:172.21.15.203:6789/0 conn(0x7fb1c80b1730 legacy=0x7fb1c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:43.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.257+0000 7fb1ce58c640 1 -- 172.21.15.143:0/1412124058 shutdown_connections 2024-03-25T19:38:43.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.257+0000 7fb1ce58c640 1 -- 172.21.15.143:0/1412124058 >> 172.21.15.143:0/1412124058 conn(0x7fb1c80617a0 msgr2=0x7fb1c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:43.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.257+0000 7fb1ce58c640 1 -- 172.21.15.143:0/1412124058 shutdown_connections 2024-03-25T19:38:43.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.257+0000 7fb1ce58c640 1 -- 172.21.15.143:0/1412124058 wait complete. 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.258+0000 7fb1ce58c640 10 monclient: build_initial_monmap 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.258+0000 7fb1ce58c640 10 monclient: monmap: 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:43.258 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:38:43.259 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:38:43.259 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:43.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.258+0000 7fb1ce58c640 1 librados: starting msgr at 2024-03-25T19:38:43.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.258+0000 7fb1ce58c640 1 librados: starting objecter 2024-03-25T19:38:43.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.258+0000 7fb1ce58c640 1 Processor -- start 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 1 -- start start 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 1 librados: setting wanted keys 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 1 librados: calling monclient init 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 10 monclient: init 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 10 monclient(hunting): picked mon.noname-b con 0x7fb1c80b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 10 monclient(hunting): picked mon.noname-c con 0x7fb1c80b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:43.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 10 monclient(hunting): picked mon.noname-a con 0x7fb1c814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:43.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c81d77d0 con 0x7fb1c814be90 2024-03-25T19:38:43.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c8138040 con 0x7fb1c80b1730 2024-03-25T19:38:43.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1c8137180 con 0x7fb1c80b4d60 2024-03-25T19:38:43.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:43.261 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.259+0000 7fb1ce58c640 10 monclient(hunting): authenticate will time out at 6539.603468s 2024-03-25T19:38:43.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.267+0000 7fb1c77fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fb1c80b4d60 0x7fb1c81d7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:57546/0 (socket says 172.21.15.143:57546) 2024-03-25T19:38:43.266 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.267+0000 7fb1c77fe640 1 -- 172.21.15.143:0/3200540814 learned_addr learned my addr 172.21.15.143:0/3200540814 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:43.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.268+0000 7fb1c4ff9640 10 client.?.objecter ms_handle_connect 0x7fb1c80b4d60 2024-03-25T19:38:43.267 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.268+0000 7fb1c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:43.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.269+0000 7fb1c4ff9640 10 client.?.objecter ms_handle_connect 0x7fb1c80b1730 2024-03-25T19:38:43.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.269+0000 7fb1c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:43.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.271+0000 7fb1c4ff9640 10 client.?.objecter ms_handle_connect 0x7fb1c814be90 2024-03-25T19:38:43.271 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.271+0000 7fb1c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:43.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.282+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369902231 0 0) 0x7fb1c8137180 con 0x7fb1c80b4d60 2024-03-25T19:38:43.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.282+0000 7fb1c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:43.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.282+0000 7fb1c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:43.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.282+0000 7fb1c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:43.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.282+0000 7fb1c4ff9640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:38:43.282 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.282+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b4004eb0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.283+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983099856 0 0) 0x7fb1c8138040 con 0x7fb1c80b1730 2024-03-25T19:38:43.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.283+0000 7fb1c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:43.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.283+0000 7fb1c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:43.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.283+0000 7fb1c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:43.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.283+0000 7fb1c4ff9640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:38:43.283 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.283+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8137180 con 0x7fb1c80b1730 2024-03-25T19:38:43.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.284+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296474412 0 0) 0x7fb1c81d77d0 con 0x7fb1c814be90 2024-03-25T19:38:43.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.284+0000 7fb1c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:38:43.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.284+0000 7fb1c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:43.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.284+0000 7fb1c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:43.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.284+0000 7fb1c4ff9640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:38:43.284 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.284+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1c8138040 con 0x7fb1c814be90 2024-03-25T19:38:43.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.291+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3819433630 0 0) 0x7fb1b4004eb0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.291+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1c81d77d0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.291+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb1bc001ce0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.291+0000 7fb1c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:43.291 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.291+0000 7fb1c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.291+0000 7fb1c4ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:43.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.291+0000 7fb1c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:43.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.295+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 205651071 0 0) 0x7fb1c8138040 con 0x7fb1c814be90 2024-03-25T19:38:43.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.296+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1b4004eb0 con 0x7fb1c814be90 2024-03-25T19:38:43.295 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.296+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb1c0001a90 con 0x7fb1c814be90 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.296+0000 7fb1c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.296+0000 7fb1c4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.296+0000 7fb1c4ff9640 10 monclient(hunting): dump: 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:43.296 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2043739266 0 0) 0x7fb1c81d77d0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 >> v1:172.21.15.203:6789/0 conn(0x7fb1c80b1730 legacy=0x7fb1c8136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 >> v1:172.21.15.143:6789/0 conn(0x7fb1c814be90 legacy=0x7fb1c81425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 1 monclient: found mon.c 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c8143900 con 0x7fb1c80b4d60 2024-03-25T19:38:43.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 5 monclient: authenticate success, global_id 4610 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 10 monclient: _renew_subs 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1c8143af0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 10 monclient: _renew_subs 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c81d85a0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.306+0000 7fb1ce58c640 1 librados: init done 2024-03-25T19:38:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.307+0000 7fb1ce58c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.307+0000 7fb1ce58c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.307+0000 7fb1ce58c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:43.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.307+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1c81401f0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.612+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1bc004a90 con 0x7fb1c80b4d60 2024-03-25T19:38:43.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.612+0000 7fb1c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:43.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.612+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb1bc004e60 con 0x7fb1c80b4d60 2024-03-25T19:38:43.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.612+0000 7fb1c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:43.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.612+0000 7fb1c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:43.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.612+0000 7fb1c4ff9640 10 monclient: dump: 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:43.613 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.612+0000 7fb1c67fc640 4 set_mon_vals no callback set 2024-03-25T19:38:43.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.613+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fb1bc005080 con 0x7fb1c80b4d60 2024-03-25T19:38:43.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.614+0000 7fb1c7fff640 10 monclient: get_auth_request con 0x7fb1b406df70 auth_method 0 2024-03-25T19:38:43.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.614+0000 7fb1c4ff9640 10 client.4610.objecter ms_handle_connect 0x7fb1b406df70 2024-03-25T19:38:43.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.618+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fb1bc0549a0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.619+0000 7fb1c4ff9640 10 client.4610.objecter ms_dispatch 0x7fb1c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:43.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.619+0000 7fb1c4ff9640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:43.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.619+0000 7fb1c4ff9640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:43.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.619+0000 7fb1c4ff9640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-25T19:38:43.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.619+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1bc0504b0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.619+0000 7fb1c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:43.672 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.619+0000 7fb1c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:43.840 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.839+0000 7fb1ce58c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:43.840 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.839+0000 7fb1ce58c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:43.840 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.839+0000 7fb1ce58c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:38:43.840 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.839+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1c80627b0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.906+0000 7fb1c4ff9640 1 -- 172.21.15.143:0/3200540814 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fb1bc0523e0 con 0x7fb1c80b4d60 2024-03-25T19:38:43.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.906+0000 7fb1c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:43.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.906+0000 7fb1c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:43.907 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:38:43.907 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:43.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 10 librados: watch_flush enter 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 10 librados: watch_flush exit 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 10 client.4610.objecter successfully canceled tick 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 >> v1:172.21.15.143:6800/1292282666 conn(0x7fb1b406df70 legacy=0x7fb1b4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 10 monclient: shutdown 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 >> v1:172.21.15.143:6790/0 conn(0x7fb1c80b4d60 legacy=0x7fb1c81d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 shutdown_connections 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.908+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 >> 172.21.15.143:0/3200540814 conn(0x7fb1c80617a0 msgr2=0x7fb1c813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.909+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 shutdown_connections 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.909+0000 7fb1ce58c640 1 -- 172.21.15.143:0/3200540814 wait complete. 2024-03-25T19:38:43.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:43.909+0000 7fb1ce58c640 1 librados: shutdown 2024-03-25T19:38:43.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:50.728 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:50.842 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.842+0000 7fcafac16640 10 monclient: get_monmap_and_config 2024-03-25T19:38:50.842 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.842+0000 7fcafac16640 10 monclient: build_initial_monmap 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.842+0000 7fcafac16640 10 monclient: monmap: 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:38:50.843635+0000 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:38:50.843635+0000 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:38:50.843 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:50.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.843+0000 7fcafac16640 1 Processor -- start 2024-03-25T19:38:50.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.843+0000 7fcafac16640 1 -- start start 2024-03-25T19:38:50.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.843+0000 7fcafac16640 10 monclient: init 2024-03-25T19:38:50.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.844+0000 7fcafac16640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:50.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.844+0000 7fcafac16640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:50.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.844+0000 7fcafac16640 10 monclient(hunting): picked mon.noname-b con 0x7fcaf40a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:50.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.844+0000 7fcafac16640 10 monclient(hunting): picked mon.noname-c con 0x7fcaf413f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:50.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.845+0000 7fcafac16640 10 monclient(hunting): picked mon.noname-a con 0x7fcaf41430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:50.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.845+0000 7fcafac16640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf4142520 con 0x7fcaf41430c0 2024-03-25T19:38:50.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.845+0000 7fcafac16640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40b28a0 con 0x7fcaf413f710 2024-03-25T19:38:50.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.845+0000 7fcafac16640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40b2a30 con 0x7fcaf40a98f0 2024-03-25T19:38:50.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.845+0000 7fcafac16640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:50.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.845+0000 7fcafac16640 10 monclient(hunting): authenticate will time out at 6547.188715s 2024-03-25T19:38:50.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.850+0000 7fcaf89b2640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fcaf40a98f0 0x7fcaf40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:45650/0 (socket says 172.21.15.143:45650) 2024-03-25T19:38:50.850 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.850+0000 7fcaf89b2640 1 -- 172.21.15.143:0/2735540018 learned_addr learned my addr 172.21.15.143:0/2735540018 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:50.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.869+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883243499 0 0) 0x7fcaf40b28a0 con 0x7fcaf413f710 2024-03-25T19:38:50.869 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.869+0000 7fcaf37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:50.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.869+0000 7fcaf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:50.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.869+0000 7fcaf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:50.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.869+0000 7fcaf37fe640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:38:50.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.869+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae4003520 con 0x7fcaf413f710 2024-03-25T19:38:50.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.875+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542940769 0 0) 0x7fcaf40b2a30 con 0x7fcaf40a98f0 2024-03-25T19:38:50.874 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.875+0000 7fcaf37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:38:50.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.875+0000 7fcaf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:50.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.875+0000 7fcaf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:50.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.875+0000 7fcaf37fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:38:50.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.875+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf40b28a0 con 0x7fcaf40a98f0 2024-03-25T19:38:50.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 101827727 0 0) 0x7fcae4003520 con 0x7fcaf413f710 2024-03-25T19:38:50.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 >> v1:172.21.15.143:6790/0 conn(0x7fcaf40a98f0 legacy=0x7fcaf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:50.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 >> v1:172.21.15.143:6789/0 conn(0x7fcaf41430c0 legacy=0x7fcaf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 1 monclient: found mon.noname-c 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf40b44c0 con 0x7fcaf413f710 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 10 monclient: _finish_auth 0 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcafac16640 5 monclient: authenticate success, global_id 4627 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fcae0001690 con 0x7fcaf413f710 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcafac16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcaf37fe640 10 monclient: dump: 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:50.884 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:50.885 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.883+0000 7fcafac16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.893+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcae00017f0 con 0x7fcaf413f710 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.893+0000 7fcaf37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.893+0000 7fcaf37fe640 1 -- 172.21.15.143:0/2735540018 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fcae0001c00 con 0x7fcaf413f710 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.893+0000 7fcafac16640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.893+0000 7fcafac16640 4 set_mon_vals no callback set 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.893+0000 7fcaf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.893+0000 7fcaf37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.893+0000 7fcaf37fe640 10 monclient: dump: 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:50.893 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:50.894 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:50.894 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:50.894 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:50.894 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:50.894 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:50.894 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:50.894 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:50.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.894+0000 7fcafac16640 10 monclient: shutdown 2024-03-25T19:38:50.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.894+0000 7fcafac16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:50.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.894+0000 7fcafac16640 1 -- 172.21.15.143:0/2735540018 >> v1:172.21.15.203:6789/0 conn(0x7fcaf413f710 legacy=0x7fcaf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:50.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.894+0000 7fcafac16640 1 -- 172.21.15.143:0/2735540018 shutdown_connections 2024-03-25T19:38:50.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.894+0000 7fcafac16640 1 -- 172.21.15.143:0/2735540018 >> 172.21.15.143:0/2735540018 conn(0x7fcaf40b7ea0 msgr2=0x7fcaf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:50.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.894+0000 7fcafac16640 1 -- 172.21.15.143:0/2735540018 shutdown_connections 2024-03-25T19:38:50.897 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.894+0000 7fcafac16640 1 -- 172.21.15.143:0/2735540018 wait complete. 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.895+0000 7fcafac16640 10 monclient: build_initial_monmap 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.895+0000 7fcafac16640 10 monclient: monmap: 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:38:50.898 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:50.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.895+0000 7fcafac16640 1 librados: starting msgr at 2024-03-25T19:38:50.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.895+0000 7fcafac16640 1 librados: starting objecter 2024-03-25T19:38:50.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.895+0000 7fcafac16640 1 Processor -- start 2024-03-25T19:38:50.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.896+0000 7fcafac16640 1 -- start start 2024-03-25T19:38:50.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.896+0000 7fcafac16640 1 librados: setting wanted keys 2024-03-25T19:38:50.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.896+0000 7fcafac16640 1 librados: calling monclient init 2024-03-25T19:38:50.899 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.896+0000 7fcafac16640 10 monclient: init 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.896+0000 7fcafac16640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.896+0000 7fcafac16640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.896+0000 7fcafac16640 10 monclient(hunting): picked mon.noname-a con 0x7fcaf40a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.897+0000 7fcafac16640 10 monclient(hunting): picked mon.noname-b con 0x7fcaf413f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.897+0000 7fcafac16640 10 monclient(hunting): picked mon.noname-c con 0x7fcaf41430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.897+0000 7fcafac16640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf413b0f0 con 0x7fcaf40a98f0 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.897+0000 7fcafac16640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf413b8f0 con 0x7fcaf413f710 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.897+0000 7fcafac16640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf413bb60 con 0x7fcaf41430c0 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.897+0000 7fcafac16640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:50.900 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.897+0000 7fcafac16640 10 monclient(hunting): authenticate will time out at 6547.240788s 2024-03-25T19:38:50.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.901+0000 7fcaf89b2640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fcaf40a98f0 0x7fcaf4139f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:57616/0 (socket says 172.21.15.143:57616) 2024-03-25T19:38:50.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.901+0000 7fcaf89b2640 1 -- 172.21.15.143:0/358815204 learned_addr learned my addr 172.21.15.143:0/358815204 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:50.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.903+0000 7fcaf17fa640 10 client.?.objecter ms_handle_connect 0x7fcaf40a98f0 2024-03-25T19:38:50.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.903+0000 7fcaf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:50.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.908+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153322518 0 0) 0x7fcaf413b0f0 con 0x7fcaf40a98f0 2024-03-25T19:38:50.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.908+0000 7fcaf17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:38:50.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.908+0000 7fcaf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:50.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.908+0000 7fcaf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:50.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.908+0000 7fcaf17fa640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:38:50.908 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.908+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcad8004eb0 con 0x7fcaf40a98f0 2024-03-25T19:38:50.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.909+0000 7fcaf17fa640 10 client.?.objecter ms_handle_connect 0x7fcaf413f710 2024-03-25T19:38:50.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.909+0000 7fcaf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:50.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.916+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3203327235 0 0) 0x7fcad8004eb0 con 0x7fcaf40a98f0 2024-03-25T19:38:50.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.916+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaf413b0f0 con 0x7fcaf40a98f0 2024-03-25T19:38:50.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.917+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fcae8001a30 con 0x7fcaf40a98f0 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.917+0000 7fcaf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.917+0000 7fcaf17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.917+0000 7fcaf17fa640 10 monclient(hunting): dump: 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:50.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.917+0000 7fcaf17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:50.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.920+0000 7fcaf17fa640 10 client.?.objecter ms_handle_connect 0x7fcaf41430c0 2024-03-25T19:38:50.920 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.920+0000 7fcaf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:50.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3460191448 0 0) 0x7fcaf413b0f0 con 0x7fcaf40a98f0 2024-03-25T19:38:50.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 >> v1:172.21.15.143:6790/0 conn(0x7fcaf41430c0 legacy=0x7fcaf41e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:50.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 >> v1:172.21.15.203:6789/0 conn(0x7fcaf413f710 legacy=0x7fcaf414bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:50.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:38:50.923 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 1 monclient: found mon.a 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf413be40 con 0x7fcaf40a98f0 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 10 monclient: _finish_auth 0 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcaf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 5 monclient: authenticate success, global_id 4626 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 10 monclient: _renew_subs 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf414e670 con 0x7fcaf40a98f0 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 10 monclient: _renew_subs 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf413ca00 con 0x7fcaf40a98f0 2024-03-25T19:38:50.924 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.923+0000 7fcafac16640 1 librados: init done 2024-03-25T19:38:50.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.924+0000 7fcafac16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:50.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.924+0000 7fcafac16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:50.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.924+0000 7fcafac16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:50.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:50.924+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf40627b0 con 0x7fcaf40a98f0 2024-03-25T19:38:51.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.231+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcae8001c00 con 0x7fcaf40a98f0 2024-03-25T19:38:51.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.231+0000 7fcaf17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:51.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.231+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fcae8004cb0 con 0x7fcaf40a98f0 2024-03-25T19:38:51.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.231+0000 7fcaf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.231+0000 7fcaf17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.231+0000 7fcaf17fa640 10 monclient: dump: 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:51.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.231+0000 7fcaf2ffd640 4 set_mon_vals no callback set 2024-03-25T19:38:51.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.232+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fcae8004e90 con 0x7fcaf40a98f0 2024-03-25T19:38:51.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.233+0000 7fcaf3fff640 10 monclient: get_auth_request con 0x7fcad806dcf0 auth_method 0 2024-03-25T19:38:51.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.233+0000 7fcaf17fa640 10 client.4626.objecter ms_handle_connect 0x7fcad806dcf0 2024-03-25T19:38:51.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.237+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fcae8054a40 con 0x7fcaf40a98f0 2024-03-25T19:38:51.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.237+0000 7fcaf17fa640 10 client.4626.objecter ms_dispatch 0x7fcaf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:51.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.237+0000 7fcaf17fa640 3 client.4626.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:51.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.237+0000 7fcaf17fa640 3 client.4626.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:51.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.238+0000 7fcaf17fa640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-25T19:38:51.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.238+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcae8050380 con 0x7fcaf40a98f0 2024-03-25T19:38:51.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.238+0000 7fcaf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:51.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.238+0000 7fcaf17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:51.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.489+0000 7fcafac16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:51.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.489+0000 7fcafac16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:51.489 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.489+0000 7fcafac16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:51.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.489+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcaf41398e0 con 0x7fcaf40a98f0 2024-03-25T19:38:51.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.518+0000 7fcaf17fa640 1 -- 172.21.15.143:0/358815204 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fcae8052480 con 0x7fcaf40a98f0 2024-03-25T19:38:51.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.519+0000 7fcaf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:51.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.519+0000 7fcaf17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:51.519 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:38:51.519 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 10 librados: watch_flush enter 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 10 librados: watch_flush exit 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 10 client.4626.objecter successfully canceled tick 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 >> v1:172.21.15.143:6800/1292282666 conn(0x7fcad806dcf0 legacy=0x7fcad80701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 10 monclient: shutdown 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 >> v1:172.21.15.143:6789/0 conn(0x7fcaf40a98f0 legacy=0x7fcaf4139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 shutdown_connections 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.521+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 >> 172.21.15.143:0/358815204 conn(0x7fcaf40b7ea0 msgr2=0x7fcaf414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.522+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 shutdown_connections 2024-03-25T19:38:51.522 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.522+0000 7fcafac16640 1 -- 172.21.15.143:0/358815204 wait complete. 2024-03-25T19:38:51.523 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:51.522+0000 7fcafac16640 1 librados: shutdown 2024-03-25T19:38:51.534 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:58.341 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.455+0000 7f842f655640 10 monclient: get_monmap_and_config 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.455+0000 7f842f655640 10 monclient: build_initial_monmap 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.455+0000 7f842f655640 10 monclient: monmap: 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:38:58.456313+0000 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:38:58.456313+0000 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:58.455 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:58.456 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:38:58.456 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:38:58.456 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.456+0000 7f842f655640 1 Processor -- start 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.456+0000 7f842f655640 1 -- start start 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.456+0000 7f842f655640 10 monclient: init 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.456+0000 7f842f655640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 10 monclient(hunting): picked mon.noname-c con 0x7f84280a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 10 monclient(hunting): picked mon.noname-a con 0x7f842813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 10 monclient(hunting): picked mon.noname-b con 0x7f84281430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8428142520 con 0x7f842813f710 2024-03-25T19:38:58.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84280b28a0 con 0x7f84280a98f0 2024-03-25T19:38:58.458 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84280b2a30 con 0x7f84281430c0 2024-03-25T19:38:58.458 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:58.458 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.457+0000 7f842f655640 10 monclient(hunting): authenticate will time out at 6554.800939s 2024-03-25T19:38:58.466 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.467+0000 7f842d3f1640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f84280a98f0 0x7f84280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:59600/0 (socket says 172.21.15.143:59600) 2024-03-25T19:38:58.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.467+0000 7f842d3f1640 1 -- 172.21.15.143:0/1973202587 learned_addr learned my addr 172.21.15.143:0/1973202587 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:58.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.479+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850589147 0 0) 0x7f84280b28a0 con 0x7f84280a98f0 2024-03-25T19:38:58.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.479+0000 7f841ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:58.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.479+0000 7f841ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:58.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.479+0000 7f841ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:58.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.479+0000 7f841ffff640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:38:58.479 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.479+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8414003520 con 0x7f84280a98f0 2024-03-25T19:38:58.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.491+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092889544 0 0) 0x7f84280b2a30 con 0x7f84281430c0 2024-03-25T19:38:58.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.491+0000 7f841ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:38:58.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.491+0000 7f841ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:58.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.491+0000 7f841ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:58.490 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.491+0000 7f841ffff640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:38:58.491 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.491+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84280b28a0 con 0x7f84281430c0 2024-03-25T19:38:58.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3741266571 0 0) 0x7f8414003520 con 0x7f84280a98f0 2024-03-25T19:38:58.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 >> v1:172.21.15.143:6790/0 conn(0x7f84281430c0 legacy=0x7f842814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:58.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 >> v1:172.21.15.143:6789/0 conn(0x7f842813f710 legacy=0x7f84281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:58.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:38:58.492 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 1 monclient: found mon.noname-c 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84280b44c0 con 0x7f84280a98f0 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 10 monclient: _finish_auth 0 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8424001690 con 0x7f84280a98f0 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f842f655640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f842f655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f841ffff640 10 monclient: dump: 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.493 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:58.494 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:58.494 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:58.494 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:58.494 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:58.494 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:58.494 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.492+0000 7f842f655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.515+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8424001870 con 0x7f84280a98f0 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.515+0000 7f841ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.515+0000 7f841ffff640 1 -- 172.21.15.143:0/1973202587 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8424001c80 con 0x7f84280a98f0 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.515+0000 7f841ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.515+0000 7f841ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.515+0000 7f841ffff640 10 monclient: dump: 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.515 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.516 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:58.516 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:58.516 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:58.516 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:58.516 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:58.516 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:58.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.515+0000 7f842f655640 10 monclient: get_monmap_and_config success 2024-03-25T19:38:58.516 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.515+0000 7f842f655640 4 set_mon_vals no callback set 2024-03-25T19:38:58.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.516+0000 7f842f655640 10 monclient: shutdown 2024-03-25T19:38:58.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.516+0000 7f842f655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:58.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.516+0000 7f842f655640 1 -- 172.21.15.143:0/1973202587 >> v1:172.21.15.203:6789/0 conn(0x7f84280a98f0 legacy=0x7f84280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:58.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.516+0000 7f842f655640 1 -- 172.21.15.143:0/1973202587 shutdown_connections 2024-03-25T19:38:58.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.516+0000 7f842f655640 1 -- 172.21.15.143:0/1973202587 >> 172.21.15.143:0/1973202587 conn(0x7f84280b7ea0 msgr2=0x7f84280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:58.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.516+0000 7f842f655640 1 -- 172.21.15.143:0/1973202587 shutdown_connections 2024-03-25T19:38:58.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.516+0000 7f842f655640 1 -- 172.21.15.143:0/1973202587 wait complete. 2024-03-25T19:38:58.517 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 10 monclient: build_initial_monmap 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 10 monclient: monmap: 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 1 librados: starting msgr at 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 1 librados: starting objecter 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 1 Processor -- start 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 1 -- start start 2024-03-25T19:38:58.518 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 1 librados: setting wanted keys 2024-03-25T19:38:58.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 1 librados: calling monclient init 2024-03-25T19:38:58.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.517+0000 7f842f655640 10 monclient: init 2024-03-25T19:38:58.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 10 monclient: _reopen_session rank -1 2024-03-25T19:38:58.519 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:38:58.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 10 monclient(hunting): picked mon.noname-c con 0x7f84280a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:38:58.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 10 monclient(hunting): picked mon.noname-b con 0x7f842813f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:38:58.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 10 monclient(hunting): picked mon.noname-a con 0x7f84281430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:38:58.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8428135590 con 0x7f84281430c0 2024-03-25T19:38:58.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8428136e50 con 0x7f842813f710 2024-03-25T19:38:58.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8428135070 con 0x7f84280a98f0 2024-03-25T19:38:58.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 10 monclient(hunting): _renew_subs 2024-03-25T19:38:58.520 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.518+0000 7f842f655640 10 monclient(hunting): authenticate will time out at 6554.862447s 2024-03-25T19:38:58.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.530+0000 7f842cbf0640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f842813f710 0x7f84281d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:59646/0 (socket says 172.21.15.143:59646) 2024-03-25T19:38:58.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.530+0000 7f842cbf0640 1 -- 172.21.15.143:0/514090667 learned_addr learned my addr 172.21.15.143:0/514090667 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:38:58.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.532+0000 7f841dffb640 10 client.?.objecter ms_handle_connect 0x7f842813f710 2024-03-25T19:38:58.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.532+0000 7f841dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:58.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.532+0000 7f841dffb640 10 client.?.objecter ms_handle_connect 0x7f84280a98f0 2024-03-25T19:38:58.532 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.532+0000 7f841dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:58.543 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.544+0000 7f841dffb640 10 client.?.objecter ms_handle_connect 0x7f84281430c0 2024-03-25T19:38:58.544 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.544+0000 7f841dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:38:58.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.613+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551424621 0 0) 0x7f8428135590 con 0x7f84281430c0 2024-03-25T19:38:58.613 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.613+0000 7f841dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:58.613 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.613+0000 7f841dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:58.613 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.613+0000 7f841dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:58.613 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.613+0000 7f841dffb640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:38:58.613 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.613+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c004eb0 con 0x7f84281430c0 2024-03-25T19:38:58.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.615+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946694743 0 0) 0x7f8428136e50 con 0x7f842813f710 2024-03-25T19:38:58.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.615+0000 7f841dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:58.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.615+0000 7f841dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:58.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.615+0000 7f841dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:58.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.615+0000 7f841dffb640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:38:58.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.615+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8428135590 con 0x7f842813f710 2024-03-25T19:38:58.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.622+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059652780 0 0) 0x7f8428135070 con 0x7f84280a98f0 2024-03-25T19:38:58.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.622+0000 7f841dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:38:58.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.622+0000 7f841dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:38:58.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.622+0000 7f841dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:38:58.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.622+0000 7f841dffb640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:38:58.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.622+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8428136e50 con 0x7f84280a98f0 2024-03-25T19:38:58.640 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.641+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1835497452 0 0) 0x7f840c004eb0 con 0x7f84281430c0 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.641+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8428135070 con 0x7f84281430c0 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.641+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8420001bd0 con 0x7f84281430c0 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.641+0000 7f841dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.641+0000 7f841dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.641+0000 7f841dffb640 10 monclient(hunting): dump: 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:58.641 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:58.642 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:58.642 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:58.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.641+0000 7f841dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1605612964 0 0) 0x7f8428135590 con 0x7f842813f710 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f840c004eb0 con 0x7f842813f710 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8418001970 con 0x7f842813f710 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 10 monclient(hunting): dump: 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.643 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1164188960 0 0) 0x7f8428136e50 con 0x7f84280a98f0 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8428135590 con 0x7f84280a98f0 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8424001b80 con 0x7f84280a98f0 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.642+0000 7f841dffb640 10 monclient(hunting): dump: 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:58.644 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:58.645 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.645 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.645 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:58.645 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:58.645 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:58.645 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:58.645 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:58.645 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:58.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 804943221 0 0) 0x7f8428135070 con 0x7f84281430c0 2024-03-25T19:38:58.652 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 >> v1:172.21.15.143:6790/0 conn(0x7f84280a98f0 legacy=0x7f8428130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 >> v1:172.21.15.203:6789/0 conn(0x7f842813f710 legacy=0x7f84281d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 1 monclient: found mon.a 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8428135d80 con 0x7f84281430c0 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 10 monclient: _finish_auth 0 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f841dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f842f655640 5 monclient: authenticate success, global_id 4632 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f842f655640 10 monclient: _renew_subs 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f842f655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:58.653 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8428131660 con 0x7f84281430c0 2024-03-25T19:38:58.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.652+0000 7f842f655640 10 client.4632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:38:58.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.653+0000 7f842f655640 10 monclient: _renew_subs 2024-03-25T19:38:58.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.653+0000 7f842f655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:58.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.653+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84280b7a80 con 0x7f84281430c0 2024-03-25T19:38:58.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.653+0000 7f842f655640 1 librados: init done 2024-03-25T19:38:58.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.654+0000 7f842f655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:58.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.654+0000 7f842f655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:58.654 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.654+0000 7f842f655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:58.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.654+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8428131a00 con 0x7f84281430c0 2024-03-25T19:38:58.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.949+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84200020c0 con 0x7f84281430c0 2024-03-25T19:38:58.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.949+0000 7f841dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:38:58.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.949+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8420003970 con 0x7f84281430c0 2024-03-25T19:38:58.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.949+0000 7f841dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:38:58.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.949+0000 7f841dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:38:58.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.949+0000 7f841dffb640 10 monclient: dump: 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:38:58.950 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.949+0000 7f841f7fe640 4 set_mon_vals no callback set 2024-03-25T19:38:58.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.950+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f84200199c0 con 0x7f84281430c0 2024-03-25T19:38:58.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.950+0000 7f842d3f1640 10 monclient: get_auth_request con 0x7f840c06e0b0 auth_method 0 2024-03-25T19:38:58.951 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.950+0000 7f841dffb640 10 client.4632.objecter ms_handle_connect 0x7f840c06e0b0 2024-03-25T19:38:58.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.956+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f8420053710 con 0x7f84281430c0 2024-03-25T19:38:58.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.956+0000 7f841dffb640 10 client.4632.objecter ms_dispatch 0x7f8428000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:38:58.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.956+0000 7f841dffb640 3 client.4632.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:38:58.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.956+0000 7f841dffb640 3 client.4632.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:38:58.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.956+0000 7f841dffb640 20 client.4632.objecter dump_active .. 0 homeless 2024-03-25T19:38:58.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.956+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f842004f050 con 0x7f84281430c0 2024-03-25T19:38:58.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.956+0000 7f841dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:38:58.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:58.956+0000 7f841dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:38:59.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.211+0000 7f842f655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:59.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.211+0000 7f842f655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:59.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.211+0000 7f842f655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:38:59.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.211+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8428135200 con 0x7f84281430c0 2024-03-25T19:38:59.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.231+0000 7f841dffb640 1 -- 172.21.15.143:0/514090667 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f8420051150 con 0x7f84281430c0 2024-03-25T19:38:59.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.231+0000 7f841dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:38:59.231 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.231+0000 7f841dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:38:59.231 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:38:59.231 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 10 librados: watch_flush enter 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 10 librados: watch_flush exit 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 20 client.4632.objecter shutdown clearing up homeless session... 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 10 client.4632.objecter successfully canceled tick 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 >> v1:172.21.15.143:6800/1292282666 conn(0x7f840c06e0b0 legacy=0x7f840c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 10 monclient: shutdown 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 >> v1:172.21.15.143:6789/0 conn(0x7f84281430c0 legacy=0x7f84281e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 shutdown_connections 2024-03-25T19:38:59.234 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 >> 172.21.15.143:0/514090667 conn(0x7f84280b7ea0 msgr2=0x7f8428074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:38:59.235 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 shutdown_connections 2024-03-25T19:38:59.235 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 1 -- 172.21.15.143:0/514090667 wait complete. 2024-03-25T19:38:59.235 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:38:59.234+0000 7f842f655640 1 librados: shutdown 2024-03-25T19:38:59.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:38:59.434 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:25.998 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.481 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:26.046 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.528 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:26.093 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.529 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:26.065 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.542 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:26.077 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.544 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:26.109 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.559 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:26.124 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.571 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:26.107 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:38:59.584 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:26.149 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:06.054 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.171+0000 7f7659bb5640 10 monclient: get_monmap_and_config 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.171+0000 7f7659bb5640 10 monclient: build_initial_monmap 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.171+0000 7f7659bb5640 10 monclient: monmap: 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:39:06.172673+0000 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:39:06.172673+0000 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:06.171 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:06.172 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:06.172 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:39:06.172 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:39:06.172 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:06.172 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.172+0000 7f7659bb5640 1 Processor -- start 2024-03-25T19:39:06.172 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.172+0000 7f7659bb5640 1 -- start start 2024-03-25T19:39:06.172 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.172+0000 7f7659bb5640 10 monclient: init 2024-03-25T19:39:06.173 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 10 monclient(hunting): picked mon.noname-c con 0x7f76540a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 10 monclient(hunting): picked mon.noname-a con 0x7f765413f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 10 monclient(hunting): picked mon.noname-b con 0x7f76541430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7654142520 con 0x7f765413f710 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76540b28a0 con 0x7f76540a98f0 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76540b2a30 con 0x7f76541430c0 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:06.174 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.173+0000 7f7659bb5640 10 monclient(hunting): authenticate will time out at 6562.517533s 2024-03-25T19:39:06.186 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.186+0000 7f7652ffd640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f765413f710 0x7f76541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:53764/0 (socket says 172.21.15.143:53764) 2024-03-25T19:39:06.186 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.186+0000 7f7652ffd640 1 -- 172.21.15.143:0/569302252 learned_addr learned my addr 172.21.15.143:0/569302252 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:06.199 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.199+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550372648 0 0) 0x7f76540b28a0 con 0x7f76540a98f0 2024-03-25T19:39:06.199 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.199+0000 7f76527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:06.199 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.199+0000 7f76527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:06.199 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.199+0000 7f76527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:06.199 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.199+0000 7f76527fc640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:39:06.199 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.199+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f763c003520 con 0x7f76540a98f0 2024-03-25T19:39:06.202 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.203+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719484329 0 0) 0x7f7654142520 con 0x7f765413f710 2024-03-25T19:39:06.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.203+0000 7f76527fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:06.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.203+0000 7f76527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:06.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.203+0000 7f76527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:06.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.203+0000 7f76527fc640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:39:06.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.203+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76540b28a0 con 0x7f765413f710 2024-03-25T19:39:06.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861558045 0 0) 0x7f76540b28a0 con 0x7f765413f710 2024-03-25T19:39:06.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 >> v1:172.21.15.143:6790/0 conn(0x7f76541430c0 legacy=0x7f765414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:06.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 >> v1:172.21.15.203:6789/0 conn(0x7f76540a98f0 legacy=0x7f76540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:06.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:06.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 1 monclient: found mon.noname-a 2024-03-25T19:39:06.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:06.211 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76540b44c0 con 0x7f765413f710 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7648001670 con 0x7f765413f710 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f76527fc640 10 monclient: dump: 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:06.212 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:06.213 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:06.213 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:06.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f7659bb5640 5 monclient: authenticate success, global_id 4635 2024-03-25T19:39:06.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.211+0000 7f7659bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.217+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76480017d0 con 0x7f765413f710 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.217+0000 7f76527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.217+0000 7f76527fc640 1 -- 172.21.15.143:0/569302252 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7648001ba0 con 0x7f765413f710 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.217+0000 7f7659bb5640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.217+0000 7f7659bb5640 4 set_mon_vals no callback set 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.217+0000 7f76527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.217+0000 7f76527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.217+0000 7f76527fc640 10 monclient: dump: 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:06.217 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:06.218 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:06.218 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:06.218 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:06.218 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:06.218 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:06.218 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.218+0000 7f7659bb5640 10 monclient: shutdown 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.218+0000 7f7659bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.218+0000 7f7659bb5640 1 -- 172.21.15.143:0/569302252 >> v1:172.21.15.143:6789/0 conn(0x7f765413f710 legacy=0x7f76541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.218+0000 7f7659bb5640 1 -- 172.21.15.143:0/569302252 shutdown_connections 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.218+0000 7f7659bb5640 1 -- 172.21.15.143:0/569302252 >> 172.21.15.143:0/569302252 conn(0x7f76540b7ea0 msgr2=0x7f76540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.218+0000 7f7659bb5640 1 -- 172.21.15.143:0/569302252 shutdown_connections 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.218+0000 7f7659bb5640 1 -- 172.21.15.143:0/569302252 wait complete. 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 10 monclient: build_initial_monmap 2024-03-25T19:39:06.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 10 monclient: monmap: 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 1 librados: starting msgr at 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 1 librados: starting objecter 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 1 Processor -- start 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 1 -- start start 2024-03-25T19:39:06.221 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 1 librados: setting wanted keys 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 1 librados: calling monclient init 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.219+0000 7f7659bb5640 10 monclient: init 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 10 monclient(hunting): picked mon.noname-b con 0x7f76540a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 10 monclient(hunting): picked mon.noname-a con 0x7f765413f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 10 monclient(hunting): picked mon.noname-c con 0x7f76541430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76540abc00 con 0x7f765413f710 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76541d51f0 con 0x7f76540a98f0 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7654138c70 con 0x7f76541430c0 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:06.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.220+0000 7f7659bb5640 10 monclient(hunting): authenticate will time out at 6562.564162s 2024-03-25T19:39:06.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.224+0000 7f7653fff640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f76541430c0 0x7f76541e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:36872/0 (socket says 172.21.15.143:36872) 2024-03-25T19:39:06.223 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.224+0000 7f7653fff640 1 -- 172.21.15.143:0/1655261905 learned_addr learned my addr 172.21.15.143:0/1655261905 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:06.226 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.227+0000 7f763bfff640 10 client.?.objecter ms_handle_connect 0x7f76541430c0 2024-03-25T19:39:06.227 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.227+0000 7f763bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:06.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.230+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102088054 0 0) 0x7f7654138c70 con 0x7f76541430c0 2024-03-25T19:39:06.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.230+0000 7f763bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:06.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.230+0000 7f763bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:06.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.230+0000 7f763bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:06.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.230+0000 7f763bfff640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:39:06.230 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.230+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7640004eb0 con 0x7f76541430c0 2024-03-25T19:39:06.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.232+0000 7f763bfff640 10 client.?.objecter ms_handle_connect 0x7f76540a98f0 2024-03-25T19:39:06.232 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.232+0000 7f763bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:06.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1222103560 0 0) 0x7f7640004eb0 con 0x7f76541430c0 2024-03-25T19:39:06.236 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7654138c70 con 0x7f76541430c0 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f764c001c90 con 0x7f76541430c0 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 10 monclient(hunting): dump: 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:06.237 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:06.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 10 client.?.objecter ms_handle_connect 0x7f765413f710 2024-03-25T19:39:06.238 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.237+0000 7f763bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:06.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.243+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355746495 0 0) 0x7f76541d51f0 con 0x7f76540a98f0 2024-03-25T19:39:06.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.243+0000 7f763bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:06.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.243+0000 7f763bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:06.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.243+0000 7f763bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:06.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.243+0000 7f763bfff640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:39:06.243 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.243+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7640004eb0 con 0x7f76540a98f0 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 906885372 0 0) 0x7f7654138c70 con 0x7f76541430c0 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 >> v1:172.21.15.203:6789/0 conn(0x7f76540a98f0 legacy=0x7f7654138560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 >> v1:172.21.15.143:6789/0 conn(0x7f765413f710 legacy=0x7f76541d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 1 monclient: found mon.c 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76541d7730 con 0x7f76541430c0 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 10 monclient: _finish_auth 0 2024-03-25T19:39:06.245 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f763bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 5 monclient: authenticate success, global_id 4628 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 10 monclient: _renew_subs 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76541d8340 con 0x7f76541430c0 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 10 monclient: _renew_subs 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7654137f00 con 0x7f76541430c0 2024-03-25T19:39:06.246 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.245+0000 7f7659bb5640 1 librados: init done 2024-03-25T19:39:06.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.246+0000 7f7659bb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:06.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.246+0000 7f7659bb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:06.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.246+0000 7f7659bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:06.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.246+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76540627b0 con 0x7f76541430c0 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.547+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f764c001fd0 con 0x7f76541430c0 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.547+0000 7f763bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.547+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f764c003de0 con 0x7f76541430c0 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.547+0000 7f763bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.547+0000 7f763bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.547+0000 7f763bfff640 10 monclient: dump: 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:06.547 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:06.548 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:06.548 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:06.548 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:06.548 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:06.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.547+0000 7f7651ffb640 4 set_mon_vals no callback set 2024-03-25T19:39:06.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.548+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f764c004000 con 0x7f76541430c0 2024-03-25T19:39:06.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.548+0000 7f76537fe640 10 monclient: get_auth_request con 0x7f764006dd40 auth_method 0 2024-03-25T19:39:06.548 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.548+0000 7f763bfff640 10 client.4628.objecter ms_handle_connect 0x7f764006dd40 2024-03-25T19:39:06.557 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.558+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f764c054c00 con 0x7f76541430c0 2024-03-25T19:39:06.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.558+0000 7f763bfff640 10 client.4628.objecter ms_dispatch 0x7f7654000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:06.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.558+0000 7f763bfff640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:06.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.558+0000 7f763bfff640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:06.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.558+0000 7f763bfff640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T19:39:06.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.558+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f764c050710 con 0x7f76541430c0 2024-03-25T19:39:06.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.558+0000 7f763bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:06.558 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.558+0000 7f763bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:06.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.778+0000 7f7659bb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:06.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.778+0000 7f7659bb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:06.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.778+0000 7f7659bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:06.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.778+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76541d78c0 con 0x7f76541430c0 2024-03-25T19:39:06.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.798+0000 7f763bfff640 1 -- 172.21.15.143:0/1655261905 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f764c052640 con 0x7f76541430c0 2024-03-25T19:39:06.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.798+0000 7f763bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:06.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.798+0000 7f763bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:06.798 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:39:06.798 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.800+0000 7f7659bb5640 10 librados: watch_flush enter 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.800+0000 7f7659bb5640 10 librados: watch_flush exit 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.800+0000 7f7659bb5640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.800+0000 7f7659bb5640 10 client.4628.objecter successfully canceled tick 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 >> v1:172.21.15.143:6800/1292282666 conn(0x7f764006dd40 legacy=0x7f7640070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 10 monclient: shutdown 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 >> v1:172.21.15.143:6790/0 conn(0x7f76541430c0 legacy=0x7f76541e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 shutdown_connections 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 >> 172.21.15.143:0/1655261905 conn(0x7f76540b7ea0 msgr2=0x7f7654074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:06.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 shutdown_connections 2024-03-25T19:39:06.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 1 -- 172.21.15.143:0/1655261905 wait complete. 2024-03-25T19:39:06.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:06.801+0000 7f7659bb5640 1 librados: shutdown 2024-03-25T19:39:06.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:13.620 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:13.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.735+0000 7f0aefce0640 10 monclient: get_monmap_and_config 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.735+0000 7f0aefce0640 10 monclient: build_initial_monmap 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.735+0000 7f0aefce0640 10 monclient: monmap: 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:39:13.736071+0000 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:39:13.736071+0000 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:39:13.735 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.735+0000 7f0aefce0640 1 Processor -- start 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.736+0000 7f0aefce0640 1 -- start start 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.736+0000 7f0aefce0640 10 monclient: init 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.736+0000 7f0aefce0640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.736+0000 7f0aefce0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.736+0000 7f0aefce0640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae80a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.736+0000 7f0aefce0640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.737+0000 7f0aefce0640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.737+0000 7f0aefce0640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae8142520 con 0x7f0ae80a98f0 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.737+0000 7f0aefce0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae80b28a0 con 0x7f0ae81430c0 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.737+0000 7f0aefce0640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae80b2a30 con 0x7f0ae813f710 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.737+0000 7f0aefce0640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:13.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.737+0000 7f0aefce0640 10 monclient(hunting): authenticate will time out at 6570.080828s 2024-03-25T19:39:13.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.748+0000 7f0aed27b640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f0ae813f710 0x7f0ae81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:54956/0 (socket says 172.21.15.143:54956) 2024-03-25T19:39:13.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.748+0000 7f0aed27b640 1 -- 172.21.15.143:0/3095542464 learned_addr learned my addr 172.21.15.143:0/3095542464 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:13.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.753+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118906698 0 0) 0x7f0ae80b2a30 con 0x7f0ae813f710 2024-03-25T19:39:13.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.754+0000 7f0aeca7a640 10 monclient(hunting): _negotiate 2024-03-25T19:39:13.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.754+0000 7f0aeca7a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:13.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.754+0000 7f0aeca7a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:13.754 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.754+0000 7f0aeca7a640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:39:13.754 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.754+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad4003520 con 0x7f0ae813f710 2024-03-25T19:39:13.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2183430516 0 0) 0x7f0ad4003520 con 0x7f0ae813f710 2024-03-25T19:39:13.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 >> v1:172.21.15.203:6789/0 conn(0x7f0ae81430c0 legacy=0x7f0ae814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:13.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 >> v1:172.21.15.143:6789/0 conn(0x7f0ae80a98f0 legacy=0x7f0ae80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:39:13.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 10 monclient: _finish_hunting 0 2024-03-25T19:39:13.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 1 monclient: found mon.noname-b 2024-03-25T19:39:13.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:13.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae80b44c0 con 0x7f0ae813f710 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 10 monclient: _finish_auth 0 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 10 monclient: _check_auth_tickets 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0ad8001690 con 0x7f0ae813f710 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aeca7a640 10 monclient: dump: 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:13.765 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:13.766 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:13.766 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:13.766 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aefce0640 5 monclient: authenticate success, global_id 4631 2024-03-25T19:39:13.766 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.764+0000 7f0aefce0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.781+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ad8001860 con 0x7f0ae813f710 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.781+0000 7f0aeca7a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.781+0000 7f0aeca7a640 1 -- 172.21.15.143:0/3095542464 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0ad8001cb0 con 0x7f0ae813f710 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.781+0000 7f0aefce0640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.781+0000 7f0aefce0640 4 set_mon_vals no callback set 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.781+0000 7f0aeca7a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.781+0000 7f0aeca7a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.781+0000 7f0aeca7a640 10 monclient: dump: 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:13.781 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.782 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.782 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:13.782 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:13.782 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:13.782 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:13.782 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:13.782 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.782+0000 7f0aefce0640 10 monclient: shutdown 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.782+0000 7f0aefce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.782+0000 7f0aefce0640 1 -- 172.21.15.143:0/3095542464 >> v1:172.21.15.143:6790/0 conn(0x7f0ae813f710 legacy=0x7f0ae81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.782+0000 7f0aefce0640 1 -- 172.21.15.143:0/3095542464 shutdown_connections 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.782+0000 7f0aefce0640 1 -- 172.21.15.143:0/3095542464 >> 172.21.15.143:0/3095542464 conn(0x7f0ae80b7ea0 msgr2=0x7f0ae80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.782+0000 7f0aefce0640 1 -- 172.21.15.143:0/3095542464 shutdown_connections 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.782+0000 7f0aefce0640 1 -- 172.21.15.143:0/3095542464 wait complete. 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.783+0000 7f0aefce0640 10 monclient: build_initial_monmap 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.783+0000 7f0aefce0640 10 monclient: monmap: 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:39:13.783 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:39:13.784 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:13.784 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:39:13.784 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:13.784 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:39:13.784 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:39:13.784 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:13.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 librados: starting msgr at 2024-03-25T19:39:13.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 librados: starting objecter 2024-03-25T19:39:13.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 Processor -- start 2024-03-25T19:39:13.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 -- start start 2024-03-25T19:39:13.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 librados: setting wanted keys 2024-03-25T19:39:13.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 librados: calling monclient init 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 10 monclient: init 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae81430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae81d51f0 con 0x7f0ae813f710 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae8139040 con 0x7f0ae80a98f0 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae81dad70 con 0x7f0ae81430c0 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:13.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.784+0000 7f0aefce0640 10 monclient(hunting): authenticate will time out at 6570.128600s 2024-03-25T19:39:13.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.799+0000 7f0aeda7c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0ae80a98f0 0x7f0ae8138930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:59546/0 (socket says 172.21.15.143:59546) 2024-03-25T19:39:13.798 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.799+0000 7f0aeda7c640 1 -- 172.21.15.143:0/1839761603 learned_addr learned my addr 172.21.15.143:0/1839761603 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:13.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.800+0000 7f0ade7fc640 10 client.?.objecter ms_handle_connect 0x7f0ae81430c0 2024-03-25T19:39:13.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.800+0000 7f0ade7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:13.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.801+0000 7f0ade7fc640 10 client.?.objecter ms_handle_connect 0x7f0ae80a98f0 2024-03-25T19:39:13.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.801+0000 7f0ade7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:13.807 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.807+0000 7f0ade7fc640 10 client.?.objecter ms_handle_connect 0x7f0ae813f710 2024-03-25T19:39:13.807 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.807+0000 7f0ade7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:13.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580637168 0 0) 0x7f0ae81d51f0 con 0x7f0ae813f710 2024-03-25T19:39:13.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:13.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:13.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:13.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:39:13.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0acc004eb0 con 0x7f0ae813f710 2024-03-25T19:39:13.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743363950 0 0) 0x7f0ae81dad70 con 0x7f0ae81430c0 2024-03-25T19:39:13.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:13.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:13.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:13.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:39:13.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.813+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae81d51f0 con 0x7f0ae81430c0 2024-03-25T19:39:13.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.818+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198782133 0 0) 0x7f0ae8139040 con 0x7f0ae80a98f0 2024-03-25T19:39:13.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.818+0000 7f0ade7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:13.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.818+0000 7f0ade7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:13.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.818+0000 7f0ade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:13.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.818+0000 7f0ade7fc640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:39:13.818 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.818+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae81dad70 con 0x7f0ae80a98f0 2024-03-25T19:39:13.827 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.827+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 805792008 0 0) 0x7f0acc004eb0 con 0x7f0ae813f710 2024-03-25T19:39:13.827 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.827+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ae8139040 con 0x7f0ae813f710 2024-03-25T19:39:13.827 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.827+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0ad8001b70 con 0x7f0ae813f710 2024-03-25T19:39:13.827 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.827+0000 7f0ade7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:13.827 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.827+0000 7f0ade7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:13.827 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.827+0000 7f0ade7fc640 10 monclient(hunting): dump: 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:13.828 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.827+0000 7f0ade7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.829+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865724320 0 0) 0x7f0ae81d51f0 con 0x7f0ae81430c0 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.829+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0acc004eb0 con 0x7f0ae81430c0 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.829+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0ae0001b10 con 0x7f0ae81430c0 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.829+0000 7f0ade7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.829+0000 7f0ade7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.829+0000 7f0ade7fc640 10 monclient(hunting): dump: 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:13.829 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.830 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.830 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:13.830 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:13.830 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:13.830 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:13.830 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:13.830 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.830+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1952932466 0 0) 0x7f0ae81dad70 con 0x7f0ae80a98f0 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.830+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ae81d51f0 con 0x7f0ae80a98f0 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.830+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0ae40018a0 con 0x7f0ae80a98f0 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.830+0000 7f0ade7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.830+0000 7f0ade7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.830+0000 7f0ade7fc640 10 monclient(hunting): dump: 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:13.831 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:13.832 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:13.843 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2738043441 0 0) 0x7f0ae81d51f0 con 0x7f0ae80a98f0 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 >> v1:172.21.15.143:6790/0 conn(0x7f0ae81430c0 legacy=0x7f0ae81e1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 >> v1:172.21.15.143:6789/0 conn(0x7f0ae813f710 legacy=0x7f0ae81db290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 1 monclient: found mon.b 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae81db9a0 con 0x7f0ae80a98f0 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0ade7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0aefce0640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0aefce0640 10 monclient: _renew_subs 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0aefce0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae8138150 con 0x7f0ae80a98f0 2024-03-25T19:39:13.844 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0aefce0640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:13.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0aefce0640 10 monclient: _renew_subs 2024-03-25T19:39:13.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0aefce0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:13.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.843+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae81e3610 con 0x7f0ae80a98f0 2024-03-25T19:39:13.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.844+0000 7f0aefce0640 1 librados: init done 2024-03-25T19:39:13.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.845+0000 7f0aefce0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:13.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.845+0000 7f0aefce0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:13.845 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.845+0000 7f0aefce0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:13.846 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:13.845+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae80ac050 con 0x7f0ae80a98f0 2024-03-25T19:39:14.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.120+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ae40035f0 con 0x7f0ae80a98f0 2024-03-25T19:39:14.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.120+0000 7f0ade7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:14.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.120+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f0ae40037a0 con 0x7f0ae80a98f0 2024-03-25T19:39:14.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.120+0000 7f0ade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:14.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.120+0000 7f0ade7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:14.120 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.120+0000 7f0ade7fc640 10 monclient: dump: 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:14.121 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.120+0000 7f0adffff640 4 set_mon_vals no callback set 2024-03-25T19:39:14.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.121+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f0ae40039c0 con 0x7f0ae80a98f0 2024-03-25T19:39:14.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.122+0000 7f0aed27b640 10 monclient: get_auth_request con 0x7f0acc06e110 auth_method 0 2024-03-25T19:39:14.122 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.122+0000 7f0ade7fc640 10 client.4648.objecter ms_handle_connect 0x7f0acc06e110 2024-03-25T19:39:14.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.133+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f0ae4054520 con 0x7f0ae80a98f0 2024-03-25T19:39:14.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.133+0000 7f0ade7fc640 10 client.4648.objecter ms_dispatch 0x7f0ae8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:14.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.133+0000 7f0ade7fc640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:14.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.133+0000 7f0ade7fc640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:14.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.133+0000 7f0ade7fc640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T19:39:14.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.134+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ae4050030 con 0x7f0ae80a98f0 2024-03-25T19:39:14.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.134+0000 7f0ade7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:14.134 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.134+0000 7f0ade7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:14.391 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.392+0000 7f0aefce0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:14.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.392+0000 7f0aefce0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:14.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.392+0000 7f0aefce0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:14.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.392+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ae81dab60 con 0x7f0ae80a98f0 2024-03-25T19:39:14.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.430+0000 7f0ade7fc640 1 -- 172.21.15.143:0/1839761603 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f0ae4051f60 con 0x7f0ae80a98f0 2024-03-25T19:39:14.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.430+0000 7f0ade7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:14.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.430+0000 7f0ade7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:14.431 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:39:14.431 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.433+0000 7f0aefce0640 10 librados: watch_flush enter 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.433+0000 7f0aefce0640 10 librados: watch_flush exit 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.433+0000 7f0aefce0640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.433+0000 7f0aefce0640 10 client.4648.objecter successfully canceled tick 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 >> v1:172.21.15.143:6800/1292282666 conn(0x7f0acc06e110 legacy=0x7f0acc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 10 monclient: shutdown 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 >> v1:172.21.15.203:6789/0 conn(0x7f0ae80a98f0 legacy=0x7f0ae8138930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 shutdown_connections 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 >> 172.21.15.143:0/1839761603 conn(0x7f0ae80b7ea0 msgr2=0x7f0ae8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 shutdown_connections 2024-03-25T19:39:14.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 1 -- 172.21.15.143:0/1839761603 wait complete. 2024-03-25T19:39:14.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:14.434+0000 7f0aefce0640 1 librados: shutdown 2024-03-25T19:39:14.445 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:19.511 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:46.076 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.558 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:46.123 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.591 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:09:46.156 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.606 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:46.142 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.615 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:46.151 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.628 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:46.193 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.634 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:46.199 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.644 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:09:46.180 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:19.661 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:09:46.226 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:21.252 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:21.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.364+0000 7f1d74f77640 10 monclient: get_monmap_and_config 2024-03-25T19:39:21.364 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.364+0000 7f1d74f77640 10 monclient: build_initial_monmap 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.365+0000 7f1d74f77640 10 monclient: monmap: 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:39:21.365883+0000 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:39:21.365883+0000 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:39:21.365 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:21.366 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.365+0000 7f1d74f77640 1 Processor -- start 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.365+0000 7f1d74f77640 1 -- start start 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.365+0000 7f1d74f77640 10 monclient: init 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 10 monclient(hunting): picked mon.noname-c con 0x7f1d700b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 10 monclient(hunting): picked mon.noname-a con 0x7f1d700b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 10 monclient(hunting): picked mon.noname-b con 0x7f1d7014be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d700af940 con 0x7f1d700b1730 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d700b0a70 con 0x7f1d700b4d60 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d700b0d50 con 0x7f1d7014be90 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:21.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.366+0000 7f1d74f77640 10 monclient(hunting): authenticate will time out at 6577.710485s 2024-03-25T19:39:21.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.374+0000 7f1d6dd9b640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1d700b1730 0x7f1d700b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:39912/0 (socket says 172.21.15.143:39912) 2024-03-25T19:39:21.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.374+0000 7f1d6dd9b640 1 -- 172.21.15.143:0/267968712 learned_addr learned my addr 172.21.15.143:0/267968712 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:21.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.390+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502942377 0 0) 0x7f1d700b0a70 con 0x7f1d700b4d60 2024-03-25T19:39:21.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.390+0000 7f1d6d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:39:21.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.390+0000 7f1d6d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:21.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.390+0000 7f1d6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:21.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.390+0000 7f1d6d59a640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:39:21.390 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.390+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d58003520 con 0x7f1d700b4d60 2024-03-25T19:39:21.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.392+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930262586 0 0) 0x7f1d700af940 con 0x7f1d700b1730 2024-03-25T19:39:21.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.392+0000 7f1d6d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:39:21.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.392+0000 7f1d6d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:21.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.392+0000 7f1d6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:21.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.392+0000 7f1d6d59a640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:39:21.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.392+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d700b0a70 con 0x7f1d700b1730 2024-03-25T19:39:21.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.396+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812527211 0 0) 0x7f1d700b0d50 con 0x7f1d7014be90 2024-03-25T19:39:21.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.396+0000 7f1d6d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:39:21.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.396+0000 7f1d6d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:21.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.396+0000 7f1d6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:21.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.396+0000 7f1d6d59a640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:39:21.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.396+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d700af940 con 0x7f1d7014be90 2024-03-25T19:39:21.400 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336467192 0 0) 0x7f1d58003520 con 0x7f1d700b4d60 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 >> v1:172.21.15.143:6790/0 conn(0x7f1d7014be90 legacy=0x7f1d7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 >> v1:172.21.15.143:6789/0 conn(0x7f1d700b1730 legacy=0x7f1d700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 1 monclient: found mon.noname-c 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d700b1030 con 0x7f1d700b4d60 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 10 monclient: _finish_auth 0 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d60001690 con 0x7f1d700b4d60 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d6d59a640 10 monclient: dump: 2024-03-25T19:39:21.401 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d74f77640 5 monclient: authenticate success, global_id 4651 2024-03-25T19:39:21.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.400+0000 7f1d74f77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.424+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d600017f0 con 0x7f1d700b4d60 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.424+0000 7f1d6d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.424+0000 7f1d6d59a640 1 -- 172.21.15.143:0/267968712 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d60001c00 con 0x7f1d700b4d60 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.424+0000 7f1d6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.424+0000 7f1d6d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.424+0000 7f1d6d59a640 10 monclient: dump: 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.424 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.425 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:21.425 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:21.425 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:21.425 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:21.425 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:21.425 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:21.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.424+0000 7f1d74f77640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:21.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.424+0000 7f1d74f77640 4 set_mon_vals no callback set 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.425+0000 7f1d74f77640 10 monclient: shutdown 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.425+0000 7f1d74f77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.425+0000 7f1d74f77640 1 -- 172.21.15.143:0/267968712 >> v1:172.21.15.203:6789/0 conn(0x7f1d700b4d60 legacy=0x7f1d7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.425+0000 7f1d74f77640 1 -- 172.21.15.143:0/267968712 shutdown_connections 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.425+0000 7f1d74f77640 1 -- 172.21.15.143:0/267968712 >> 172.21.15.143:0/267968712 conn(0x7f1d700617a0 msgr2=0x7f1d70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.425+0000 7f1d74f77640 1 -- 172.21.15.143:0/267968712 shutdown_connections 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.425+0000 7f1d74f77640 1 -- 172.21.15.143:0/267968712 wait complete. 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 10 monclient: build_initial_monmap 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 10 monclient: monmap: 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:21.427 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 1 librados: starting msgr at 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 1 librados: starting objecter 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 1 Processor -- start 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 1 -- start start 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 1 librados: setting wanted keys 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 1 librados: calling monclient init 2024-03-25T19:39:21.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 10 monclient: init 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 10 monclient(hunting): picked mon.noname-c con 0x7f1d700b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.427+0000 7f1d74f77640 10 monclient(hunting): picked mon.noname-a con 0x7f1d700b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.428+0000 7f1d74f77640 10 monclient(hunting): picked mon.noname-b con 0x7f1d7014be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.428+0000 7f1d74f77640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d70142050 con 0x7f1d700b4d60 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.428+0000 7f1d74f77640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d701dac50 con 0x7f1d7014be90 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.428+0000 7f1d74f77640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d70140d00 con 0x7f1d700b1730 2024-03-25T19:39:21.430 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.428+0000 7f1d74f77640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:21.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.428+0000 7f1d74f77640 10 monclient(hunting): authenticate will time out at 6577.771803s 2024-03-25T19:39:21.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.438+0000 7f1d6dd9b640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f1d700b4d60 0x7f1d701db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:39984/0 (socket says 172.21.15.143:39984) 2024-03-25T19:39:21.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.438+0000 7f1d6dd9b640 1 -- 172.21.15.143:0/361851360 learned_addr learned my addr 172.21.15.143:0/361851360 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:21.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.439+0000 7f1d5effd640 10 client.?.objecter ms_handle_connect 0x7f1d700b4d60 2024-03-25T19:39:21.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.439+0000 7f1d5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:21.444 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.444+0000 7f1d5effd640 10 client.?.objecter ms_handle_connect 0x7f1d700b1730 2024-03-25T19:39:21.444 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.444+0000 7f1d5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:21.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.446+0000 7f1d5effd640 10 client.?.objecter ms_handle_connect 0x7f1d7014be90 2024-03-25T19:39:21.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.446+0000 7f1d5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:21.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.452+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368481605 0 0) 0x7f1d70142050 con 0x7f1d700b4d60 2024-03-25T19:39:21.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.452+0000 7f1d5effd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:21.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.452+0000 7f1d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:21.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.452+0000 7f1d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:21.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.452+0000 7f1d5effd640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:39:21.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.452+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d50004eb0 con 0x7f1d700b4d60 2024-03-25T19:39:21.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.453+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292304249 0 0) 0x7f1d701dac50 con 0x7f1d7014be90 2024-03-25T19:39:21.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.453+0000 7f1d5effd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:21.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.453+0000 7f1d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:21.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.453+0000 7f1d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:21.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.453+0000 7f1d5effd640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:39:21.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.453+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d70142050 con 0x7f1d7014be90 2024-03-25T19:39:21.458 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.459+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225374262 0 0) 0x7f1d70140d00 con 0x7f1d700b1730 2024-03-25T19:39:21.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.459+0000 7f1d5effd640 10 monclient(hunting): _negotiate 2024-03-25T19:39:21.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.459+0000 7f1d5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:21.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.459+0000 7f1d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:21.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.459+0000 7f1d5effd640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:39:21.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.459+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d701dac50 con 0x7f1d700b1730 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.468+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 716502520 0 0) 0x7f1d701dac50 con 0x7f1d700b1730 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.468+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d70140d00 con 0x7f1d700b1730 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.468+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d60001b00 con 0x7f1d700b1730 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.468+0000 7f1d5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.468+0000 7f1d5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.468+0000 7f1d5effd640 10 monclient(hunting): dump: 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:21.468 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:21.469 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:21.469 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:21.469 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:21.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.468+0000 7f1d5effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:21.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.469+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1877454096 0 0) 0x7f1d70142050 con 0x7f1d7014be90 2024-03-25T19:39:21.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.469+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d701dac50 con 0x7f1d7014be90 2024-03-25T19:39:21.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.469+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d68001b50 con 0x7f1d7014be90 2024-03-25T19:39:21.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.469+0000 7f1d5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.469+0000 7f1d5effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.469+0000 7f1d5effd640 10 monclient(hunting): dump: 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:21.470 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.470+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 125857360 0 0) 0x7f1d50004eb0 con 0x7f1d700b4d60 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.470+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d70142050 con 0x7f1d700b4d60 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.470+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d640019f0 con 0x7f1d700b4d60 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.470+0000 7f1d5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.470+0000 7f1d5effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.470+0000 7f1d5effd640 10 monclient(hunting): dump: 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:21.471 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.472 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.472 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:21.472 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:21.472 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:21.472 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:21.472 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:21.472 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3433627166 0 0) 0x7f1d70140d00 con 0x7f1d700b1730 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 >> v1:172.21.15.203:6789/0 conn(0x7f1d7014be90 legacy=0x7f1d701e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 >> v1:172.21.15.143:6789/0 conn(0x7f1d700b4d60 legacy=0x7f1d701db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 10 monclient: _finish_hunting 0 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 1 monclient: found mon.c 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d701423e0 con 0x7f1d700b1730 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 10 monclient: _finish_auth 0 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 10 monclient: _check_auth_tickets 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 5 monclient: authenticate success, global_id 4640 2024-03-25T19:39:21.475 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 10 monclient: _renew_subs 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7013b470 con 0x7f1d700b1730 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 10 monclient: _renew_subs 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d701dba40 con 0x7f1d700b1730 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.475+0000 7f1d74f77640 1 librados: init done 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.476+0000 7f1d74f77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:21.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.476+0000 7f1d74f77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:21.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.476+0000 7f1d74f77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:21.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.476+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d701401f0 con 0x7f1d700b1730 2024-03-25T19:39:21.768 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.769+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d60001c60 con 0x7f1d700b1730 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.769+0000 7f1d5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.769+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1d60003a40 con 0x7f1d700b1730 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.769+0000 7f1d5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.769+0000 7f1d5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.769+0000 7f1d5effd640 10 monclient: dump: 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:21.769 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:21.770 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:21.770 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:21.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.769+0000 7f1d6cd99640 4 set_mon_vals no callback set 2024-03-25T19:39:21.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.770+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f1d60019a10 con 0x7f1d700b1730 2024-03-25T19:39:21.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.770+0000 7f1d6dd9b640 10 monclient: get_auth_request con 0x7f1d5006e0a0 auth_method 0 2024-03-25T19:39:21.770 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.771+0000 7f1d5effd640 10 client.4640.objecter ms_handle_connect 0x7f1d5006e0a0 2024-03-25T19:39:21.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.776+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f1d600535c0 con 0x7f1d700b1730 2024-03-25T19:39:21.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.776+0000 7f1d5effd640 10 client.4640.objecter ms_dispatch 0x7f1d70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:21.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.776+0000 7f1d5effd640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:21.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.776+0000 7f1d5effd640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:21.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.776+0000 7f1d5effd640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-25T19:39:21.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.776+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d6004f0d0 con 0x7f1d700b1730 2024-03-25T19:39:21.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.776+0000 7f1d5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:21.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:21.776+0000 7f1d5effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:22.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.006+0000 7f1d74f77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:22.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.006+0000 7f1d74f77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:22.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.006+0000 7f1d74f77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:22.006 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.006+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d70140e90 con 0x7f1d700b1730 2024-03-25T19:39:22.038 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.038+0000 7f1d5effd640 1 -- 172.21.15.143:0/361851360 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f1d60051000 con 0x7f1d700b1730 2024-03-25T19:39:22.038 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.038+0000 7f1d5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:22.038 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.038+0000 7f1d5effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:22.038 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:39:22.038 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 10 librados: watch_flush enter 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 10 librados: watch_flush exit 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 10 client.4640.objecter successfully canceled tick 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 >> v1:172.21.15.143:6800/1292282666 conn(0x7f1d5006e0a0 legacy=0x7f1d50070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 10 monclient: shutdown 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 >> v1:172.21.15.143:6790/0 conn(0x7f1d700b1730 legacy=0x7f1d701405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.040+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 shutdown_connections 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.041+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 >> 172.21.15.143:0/361851360 conn(0x7f1d700617a0 msgr2=0x7f1d700b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.041+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 shutdown_connections 2024-03-25T19:39:22.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.041+0000 7f1d74f77640 1 -- 172.21.15.143:0/361851360 wait complete. 2024-03-25T19:39:22.042 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:22.041+0000 7f1d74f77640 1 librados: shutdown 2024-03-25T19:39:22.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:28.858 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:28.975 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.975+0000 7f9bfd5cf640 10 monclient: get_monmap_and_config 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.975+0000 7f9bfd5cf640 10 monclient: build_initial_monmap 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.975+0000 7f9bfd5cf640 10 monclient: monmap: 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:39:28.976999+0000 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:39:28.976999+0000 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:39:28.976 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.975+0000 7f9bfd5cf640 1 Processor -- start 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.975+0000 7f9bfd5cf640 1 -- start start 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.975+0000 7f9bfd5cf640 10 monclient: init 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf80a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf813f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf81430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf8142520 con 0x7f9bf80a98f0 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80b28a0 con 0x7f9bf81430c0 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80b2a30 con 0x7f9bf813f710 2024-03-25T19:39:28.978 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:28.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.976+0000 7f9bfd5cf640 10 monclient(hunting): authenticate will time out at 6585.321229s 2024-03-25T19:39:28.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.982+0000 7f9bf67fc640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f9bf813f710 0x7f9bf81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:40956/0 (socket says 172.21.15.143:40956) 2024-03-25T19:39:28.983 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.982+0000 7f9bf67fc640 1 -- 172.21.15.143:0/2072708125 learned_addr learned my addr 172.21.15.143:0/2072708125 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:28.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.994+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744291138 0 0) 0x7f9bf80b2a30 con 0x7f9bf813f710 2024-03-25T19:39:28.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.994+0000 7f9bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:28.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.994+0000 7f9bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:28.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.994+0000 7f9bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:28.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.994+0000 7f9bf5ffb640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:39:28.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:28.994+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be0003520 con 0x7f9bf813f710 2024-03-25T19:39:29.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.002+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045286376 0 0) 0x7f9bf80b28a0 con 0x7f9bf81430c0 2024-03-25T19:39:29.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.002+0000 7f9bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:29.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.002+0000 7f9bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:29.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.002+0000 7f9bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:29.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.002+0000 7f9bf5ffb640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:39:29.003 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.002+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf80b2a30 con 0x7f9bf81430c0 2024-03-25T19:39:29.004 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.003+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343782395 0 0) 0x7f9bf8142520 con 0x7f9bf80a98f0 2024-03-25T19:39:29.004 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.003+0000 7f9bf5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:29.004 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.003+0000 7f9bf5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:29.004 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.003+0000 7f9bf5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:29.004 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.003+0000 7f9bf5ffb640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:39:29.004 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.003+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf80b28a0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.015+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885468960 0 0) 0x7f9bf80b2a30 con 0x7f9bf81430c0 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 >> v1:172.21.15.143:6790/0 conn(0x7f9bf813f710 legacy=0x7f9bf81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 >> v1:172.21.15.143:6789/0 conn(0x7f9bf80a98f0 legacy=0x7f9bf80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80b44c0 con 0x7f9bf81430c0 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9bf00019f0 con 0x7f9bf81430c0 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:29.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bf5ffb640 10 monclient: dump: 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bfd5cf640 5 monclient: authenticate success, global_id 4657 2024-03-25T19:39:29.018 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.016+0000 7f9bfd5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:29.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.020+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bf0001b50 con 0x7f9bf81430c0 2024-03-25T19:39:29.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.020+0000 7f9bf5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:29.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.020+0000 7f9bf5ffb640 1 -- 172.21.15.143:0/2072708125 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9bf0001f60 con 0x7f9bf81430c0 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.020+0000 7f9bfd5cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.020+0000 7f9bfd5cf640 4 set_mon_vals no callback set 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.020+0000 7f9bf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.020+0000 7f9bf5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.020+0000 7f9bf5ffb640 10 monclient: dump: 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:29.022 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.021+0000 7f9bfd5cf640 10 monclient: shutdown 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.021+0000 7f9bfd5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.021+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2072708125 >> v1:172.21.15.203:6789/0 conn(0x7f9bf81430c0 legacy=0x7f9bf814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.021+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2072708125 shutdown_connections 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.021+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2072708125 >> 172.21.15.143:0/2072708125 conn(0x7f9bf80b7ea0 msgr2=0x7f9bf80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.021+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2072708125 shutdown_connections 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.021+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2072708125 wait complete. 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.022+0000 7f9bfd5cf640 10 monclient: build_initial_monmap 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.022+0000 7f9bfd5cf640 10 monclient: monmap: 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:39:29.024 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.023+0000 7f9bfd5cf640 1 librados: starting msgr at 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.023+0000 7f9bfd5cf640 1 librados: starting objecter 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.023+0000 7f9bfd5cf640 1 Processor -- start 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.023+0000 7f9bfd5cf640 1 -- start start 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.023+0000 7f9bfd5cf640 1 librados: setting wanted keys 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.023+0000 7f9bfd5cf640 1 librados: calling monclient init 2024-03-25T19:39:29.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.023+0000 7f9bfd5cf640 10 monclient: init 2024-03-25T19:39:29.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:29.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:29.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf80a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:29.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf813f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:29.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf81430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:29.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf8137ce0 con 0x7f9bf813f710 2024-03-25T19:39:29.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf81396e0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf81399c0 con 0x7f9bf81430c0 2024-03-25T19:39:29.027 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:29.027 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.024+0000 7f9bfd5cf640 10 monclient(hunting): authenticate will time out at 6585.368992s 2024-03-25T19:39:29.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.039+0000 7f9bf6ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9bf80a98f0 0x7f9bf81da780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:44390/0 (socket says 172.21.15.143:44390) 2024-03-25T19:39:29.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.039+0000 7f9bf6ffd640 1 -- 172.21.15.143:0/2781484529 learned_addr learned my addr 172.21.15.143:0/2781484529 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:29.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.040+0000 7f9bdf7fe640 10 client.?.objecter ms_handle_connect 0x7f9bf80a98f0 2024-03-25T19:39:29.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.040+0000 7f9bdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:29.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.042+0000 7f9bdf7fe640 10 client.?.objecter ms_handle_connect 0x7f9bf81430c0 2024-03-25T19:39:29.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.043+0000 7f9bdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:29.046 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.046+0000 7f9bdf7fe640 10 client.?.objecter ms_handle_connect 0x7f9bf813f710 2024-03-25T19:39:29.046 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.046+0000 7f9bdf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:29.052 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.052+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739739635 0 0) 0x7f9bf81399c0 con 0x7f9bf81430c0 2024-03-25T19:39:29.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.052+0000 7f9bdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:29.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.052+0000 7f9bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:29.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.052+0000 7f9bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:29.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.052+0000 7f9bdf7fe640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:39:29.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.052+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be4004eb0 con 0x7f9bf81430c0 2024-03-25T19:39:29.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.053+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587318655 0 0) 0x7f9bf81396e0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.053+0000 7f9bdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:29.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.053+0000 7f9bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:29.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.053+0000 7f9bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:29.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.053+0000 7f9bdf7fe640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:39:29.054 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.053+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf81399c0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.066+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949770369 0 0) 0x7f9bf8137ce0 con 0x7f9bf813f710 2024-03-25T19:39:29.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.066+0000 7f9bdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:29.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.066+0000 7f9bdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:29.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.066+0000 7f9bdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:29.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.066+0000 7f9bdf7fe640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:39:29.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.066+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf81396e0 con 0x7f9bf813f710 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.077+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 431359188 0 0) 0x7f9bf81399c0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.077+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf8137ce0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.077+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9be80019f0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.077+0000 7f9bdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.077+0000 7f9bdf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.077+0000 7f9bdf7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:29.078 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.079 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.079 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:29.079 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:29.079 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:29.079 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:29.079 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:29.079 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:29.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.077+0000 7f9bdf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:29.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.082+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 94387693 0 0) 0x7f9be4004eb0 con 0x7f9bf81430c0 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.082+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf81399c0 con 0x7f9bf81430c0 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.082+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9bf0001ea0 con 0x7f9bf81430c0 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.082+0000 7f9bdf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.082+0000 7f9bdf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.082+0000 7f9bdf7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:29.083 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:29.084 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:29.084 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4154093294 0 0) 0x7f9bf8137ce0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 >> v1:172.21.15.143:6790/0 conn(0x7f9bf81430c0 legacy=0x7f9bf81375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 >> v1:172.21.15.143:6789/0 conn(0x7f9bf813f710 legacy=0x7f9bf8138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 1 monclient: found mon.b 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80b0fb0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.091 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 5 monclient: authenticate success, global_id 4660 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 10 monclient: _renew_subs 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf80afdc0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 10 monclient: _renew_subs 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf80b0340 con 0x7f9bf80a98f0 2024-03-25T19:39:29.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.090+0000 7f9bfd5cf640 1 librados: init done 2024-03-25T19:39:29.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.091+0000 7f9bfd5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:29.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.091+0000 7f9bfd5cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:29.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.091+0000 7f9bfd5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:29.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.091+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf80627b0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.371 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.371+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9be8003610 con 0x7f9bf80a98f0 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.371+0000 7f9bdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.371+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9be80039a0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.371+0000 7f9bdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.371+0000 7f9bdf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.371+0000 7f9bdf7fe640 10 monclient: dump: 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:29.372 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:29.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.371+0000 7f9bf57fa640 4 set_mon_vals no callback set 2024-03-25T19:39:29.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.371+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f9be8003bc0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.372+0000 7f9bf67fc640 10 monclient: get_auth_request con 0x7f9be406df70 auth_method 0 2024-03-25T19:39:29.373 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.372+0000 7f9bdf7fe640 10 client.4660.objecter ms_handle_connect 0x7f9be406df70 2024-03-25T19:39:29.377 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.376+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f9be8054740 con 0x7f9bf80a98f0 2024-03-25T19:39:29.377 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.376+0000 7f9bdf7fe640 10 client.4660.objecter ms_dispatch 0x7f9bf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:29.377 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.376+0000 7f9bdf7fe640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:29.377 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.376+0000 7f9bdf7fe640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:29.377 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.376+0000 7f9bdf7fe640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T19:39:29.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.377+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9be8050250 con 0x7f9bf80a98f0 2024-03-25T19:39:29.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.377+0000 7f9bdf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:29.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.377+0000 7f9bdf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:29.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.627+0000 7f9bfd5cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:29.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.627+0000 7f9bfd5cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:29.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.627+0000 7f9bfd5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:29.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.627+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bf80b71a0 con 0x7f9bf80a98f0 2024-03-25T19:39:29.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.657+0000 7f9bdf7fe640 1 -- 172.21.15.143:0/2781484529 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f9be8052180 con 0x7f9bf80a98f0 2024-03-25T19:39:29.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.657+0000 7f9bdf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:29.658 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.657+0000 7f9bdf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:29.658 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:39:29.659 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 10 librados: watch_flush enter 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 10 librados: watch_flush exit 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 10 client.4660.objecter successfully canceled tick 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 >> v1:172.21.15.143:6800/1292282666 conn(0x7f9be406df70 legacy=0x7f9be4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 10 monclient: shutdown 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 >> v1:172.21.15.203:6789/0 conn(0x7f9bf80a98f0 legacy=0x7f9bf81da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 shutdown_connections 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 >> 172.21.15.143:0/2781484529 conn(0x7f9bf80b7ea0 msgr2=0x7f9bf8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:29.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 shutdown_connections 2024-03-25T19:39:29.662 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 1 -- 172.21.15.143:0/2781484529 wait complete. 2024-03-25T19:39:29.662 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:29.660+0000 7f9bfd5cf640 1 librados: shutdown 2024-03-25T19:39:29.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:36.481 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:36.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.596+0000 7fdf79127640 10 monclient: get_monmap_and_config 2024-03-25T19:39:36.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.596+0000 7fdf79127640 10 monclient: build_initial_monmap 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.596+0000 7fdf79127640 10 monclient: monmap: 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:39:36.597837+0000 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:39:36.597837+0000 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:39:36.597 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.597+0000 7fdf79127640 1 Processor -- start 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.597+0000 7fdf79127640 1 -- start start 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.597+0000 7fdf79127640 10 monclient: init 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 10 monclient(hunting): picked mon.noname-a con 0x7fdf740a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 10 monclient(hunting): picked mon.noname-b con 0x7fdf7413f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 10 monclient(hunting): picked mon.noname-c con 0x7fdf741430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf74142520 con 0x7fdf740a98f0 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b28a0 con 0x7fdf741430c0 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740b2a30 con 0x7fdf7413f710 2024-03-25T19:39:36.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:36.600 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.598+0000 7fdf79127640 10 monclient(hunting): authenticate will time out at 6592.942178s 2024-03-25T19:39:36.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.610+0000 7fdf7359e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fdf741430c0 0x7fdf7414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:54044/0 (socket says 172.21.15.143:54044) 2024-03-25T19:39:36.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.610+0000 7fdf7359e640 1 -- 172.21.15.143:0/3619984883 learned_addr learned my addr 172.21.15.143:0/3619984883 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:36.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.693+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876808407 0 0) 0x7fdf74142520 con 0x7fdf740a98f0 2024-03-25T19:39:36.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.693+0000 7fdf71d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:39:36.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.693+0000 7fdf71d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:36.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.693+0000 7fdf71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:36.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.693+0000 7fdf71d9b640 10 monclient(hunting): my global_id is 4656 2024-03-25T19:39:36.693 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.693+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c003520 con 0x7fdf740a98f0 2024-03-25T19:39:36.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1421313068 0 0) 0x7fdf5c003520 con 0x7fdf740a98f0 2024-03-25T19:39:36.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 >> v1:172.21.15.143:6790/0 conn(0x7fdf7413f710 legacy=0x7fdf741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:36.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 >> v1:172.21.15.203:6789/0 conn(0x7fdf741430c0 legacy=0x7fdf7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 1 monclient: found mon.noname-a 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf740b44c0 con 0x7fdf740a98f0 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 10 monclient: _finish_auth 0 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf64001690 con 0x7fdf740a98f0 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf71d9b640 10 monclient: dump: 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:36.711 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:36.712 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:36.712 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:36.712 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:36.712 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:36.712 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:36.712 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:36.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf79127640 5 monclient: authenticate success, global_id 4656 2024-03-25T19:39:36.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.710+0000 7fdf79127640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:36.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.732+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf640017f0 con 0x7fdf740a98f0 2024-03-25T19:39:36.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.732+0000 7fdf71d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:36.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.732+0000 7fdf71d9b640 1 -- 172.21.15.143:0/3619984883 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf64001c00 con 0x7fdf740a98f0 2024-03-25T19:39:36.732 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.732+0000 7fdf79127640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.732+0000 7fdf79127640 4 set_mon_vals no callback set 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.732+0000 7fdf71d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.732+0000 7fdf71d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.732+0000 7fdf71d9b640 10 monclient: dump: 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:36.733 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:36.734 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.733+0000 7fdf79127640 10 monclient: shutdown 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.733+0000 7fdf79127640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.733+0000 7fdf79127640 1 -- 172.21.15.143:0/3619984883 >> v1:172.21.15.143:6789/0 conn(0x7fdf740a98f0 legacy=0x7fdf740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.733+0000 7fdf79127640 1 -- 172.21.15.143:0/3619984883 shutdown_connections 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.733+0000 7fdf79127640 1 -- 172.21.15.143:0/3619984883 >> 172.21.15.143:0/3619984883 conn(0x7fdf740b7ea0 msgr2=0x7fdf740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.733+0000 7fdf79127640 1 -- 172.21.15.143:0/3619984883 shutdown_connections 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.733+0000 7fdf79127640 1 -- 172.21.15.143:0/3619984883 wait complete. 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.734+0000 7fdf79127640 10 monclient: build_initial_monmap 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.734+0000 7fdf79127640 10 monclient: monmap: 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:36.736 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.735+0000 7fdf79127640 1 librados: starting msgr at 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.735+0000 7fdf79127640 1 librados: starting objecter 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.735+0000 7fdf79127640 1 Processor -- start 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.735+0000 7fdf79127640 1 -- start start 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.735+0000 7fdf79127640 1 librados: setting wanted keys 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.735+0000 7fdf79127640 1 librados: calling monclient init 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.735+0000 7fdf79127640 10 monclient: init 2024-03-25T19:39:36.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 10 monclient(hunting): picked mon.noname-c con 0x7fdf740a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 10 monclient(hunting): picked mon.noname-b con 0x7fdf7413f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 10 monclient(hunting): picked mon.noname-a con 0x7fdf741430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7414e520 con 0x7fdf741430c0 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf74139730 con 0x7fdf7413f710 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7406eeb0 con 0x7fdf740a98f0 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:36.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.736+0000 7fdf79127640 10 monclient(hunting): authenticate will time out at 6593.079858s 2024-03-25T19:39:36.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.741+0000 7fdf7259c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fdf7413f710 0x7fdf740707d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:54062/0 (socket says 172.21.15.143:54062) 2024-03-25T19:39:36.741 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.741+0000 7fdf7259c640 1 -- 172.21.15.143:0/3325821068 learned_addr learned my addr 172.21.15.143:0/3325821068 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:36.743 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.744+0000 7fdf5b7fe640 10 client.?.objecter ms_handle_connect 0x7fdf7413f710 2024-03-25T19:39:36.743 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.744+0000 7fdf5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:36.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.748+0000 7fdf5b7fe640 10 client.?.objecter ms_handle_connect 0x7fdf740a98f0 2024-03-25T19:39:36.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.748+0000 7fdf5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:36.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.752+0000 7fdf5b7fe640 10 client.?.objecter ms_handle_connect 0x7fdf741430c0 2024-03-25T19:39:36.751 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.752+0000 7fdf5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:36.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.753+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601991699 0 0) 0x7fdf74139730 con 0x7fdf7413f710 2024-03-25T19:39:36.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.753+0000 7fdf5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:36.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.753+0000 7fdf5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:36.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.753+0000 7fdf5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:36.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.753+0000 7fdf5b7fe640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:39:36.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.753+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf60004eb0 con 0x7fdf7413f710 2024-03-25T19:39:36.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.761+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723688181 0 0) 0x7fdf7406eeb0 con 0x7fdf740a98f0 2024-03-25T19:39:36.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.761+0000 7fdf5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:36.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.761+0000 7fdf5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:36.762 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.761+0000 7fdf5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:36.762 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.761+0000 7fdf5b7fe640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:39:36.762 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.761+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf74139730 con 0x7fdf740a98f0 2024-03-25T19:39:36.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.765+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882037249 0 0) 0x7fdf7414e520 con 0x7fdf741430c0 2024-03-25T19:39:36.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.765+0000 7fdf5b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:36.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.765+0000 7fdf5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:36.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.765+0000 7fdf5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:36.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.765+0000 7fdf5b7fe640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:39:36.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.765+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf7406eeb0 con 0x7fdf741430c0 2024-03-25T19:39:36.775 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.776+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 450048960 0 0) 0x7fdf74139730 con 0x7fdf740a98f0 2024-03-25T19:39:36.775 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.776+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf7414e520 con 0x7fdf740a98f0 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.776+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf64001b80 con 0x7fdf740a98f0 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.776+0000 7fdf5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.776+0000 7fdf5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.776+0000 7fdf5b7fe640 10 monclient(hunting): dump: 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:36.776 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:36.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.776+0000 7fdf5b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:36.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 289519900 0 0) 0x7fdf7414e520 con 0x7fdf740a98f0 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 >> v1:172.21.15.203:6789/0 conn(0x7fdf7413f710 legacy=0x7fdf740707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 >> v1:172.21.15.143:6789/0 conn(0x7fdf741430c0 legacy=0x7fdf741e63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 1 monclient: found mon.c 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7406fa80 con 0x7fdf740a98f0 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf79127640 5 monclient: authenticate success, global_id 4652 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.785+0000 7fdf79127640 10 monclient: _renew_subs 2024-03-25T19:39:36.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.786+0000 7fdf79127640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:36.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.786+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7406fd60 con 0x7fdf740a98f0 2024-03-25T19:39:36.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.786+0000 7fdf79127640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:36.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.786+0000 7fdf79127640 10 monclient: _renew_subs 2024-03-25T19:39:36.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.786+0000 7fdf79127640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:36.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.786+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf74070ee0 con 0x7fdf740a98f0 2024-03-25T19:39:36.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.786+0000 7fdf79127640 1 librados: init done 2024-03-25T19:39:36.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.787+0000 7fdf79127640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:36.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.787+0000 7fdf79127640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:36.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.787+0000 7fdf79127640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:36.788 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.787+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf740627b0 con 0x7fdf740a98f0 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.808+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf640038a0 con 0x7fdf740a98f0 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.808+0000 7fdf5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.808+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf64001250 con 0x7fdf740a98f0 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.808+0000 7fdf5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.808+0000 7fdf5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.808+0000 7fdf5b7fe640 10 monclient: dump: 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:36.808 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:36.809 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:36.809 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:36.809 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:36.809 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:36.809 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:36.809 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.808+0000 7fdf7159a640 4 set_mon_vals no callback set 2024-03-25T19:39:36.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.809+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fdf64019ae0 con 0x7fdf740a98f0 2024-03-25T19:39:36.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.809+0000 7fdf7259c640 10 monclient: get_auth_request con 0x7fdf6006de20 auth_method 0 2024-03-25T19:39:36.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.809+0000 7fdf5b7fe640 10 client.4652.objecter ms_handle_connect 0x7fdf6006de20 2024-03-25T19:39:36.812 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.812+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7fdf640536e0 con 0x7fdf740a98f0 2024-03-25T19:39:36.812 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.812+0000 7fdf5b7fe640 10 client.4652.objecter ms_dispatch 0x7fdf74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:36.812 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.812+0000 7fdf5b7fe640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:36.812 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.812+0000 7fdf5b7fe640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:36.812 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:36.812+0000 7fdf5b7fe640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T19:39:37.100 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.101+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf6404f020 con 0x7fdf740a98f0 2024-03-25T19:39:37.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.101+0000 7fdf5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:37.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.101+0000 7fdf5b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.351+0000 7fdf79127640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.351+0000 7fdf79127640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.351+0000 7fdf79127640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:39:37.351 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.351+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf7414e810 con 0x7fdf740a98f0 2024-03-25T19:39:37.389 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.389+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3325821068 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7fdf64051120 con 0x7fdf740a98f0 2024-03-25T19:39:37.389 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.389+0000 7fdf5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:37.389 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.389+0000 7fdf5b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:37.389 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:39:37.390 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.391+0000 7fdf79127640 10 librados: watch_flush enter 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.391+0000 7fdf79127640 10 librados: watch_flush exit 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.391+0000 7fdf79127640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.391+0000 7fdf79127640 10 client.4652.objecter successfully canceled tick 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.391+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 >> v1:172.21.15.143:6800/1292282666 conn(0x7fdf6006de20 legacy=0x7fdf600702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.392+0000 7fdf79127640 10 monclient: shutdown 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.392+0000 7fdf79127640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.392+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 >> v1:172.21.15.143:6790/0 conn(0x7fdf740a98f0 legacy=0x7fdf7414bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.392+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 shutdown_connections 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.392+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 >> 172.21.15.143:0/3325821068 conn(0x7fdf740b7ea0 msgr2=0x7fdf74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.392+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 shutdown_connections 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.392+0000 7fdf79127640 1 -- 172.21.15.143:0/3325821068 wait complete. 2024-03-25T19:39:37.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:37.392+0000 7fdf79127640 1 librados: shutdown 2024-03-25T19:39:37.404 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:39.568 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:06.133 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.615 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:06.180 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.645 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:06.210 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.663 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:06.199 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.670 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:06.206 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.675 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:06.240 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.688 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:06.253 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.707 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:06.242 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:39.724 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:06.289 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:44.210 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:44.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.329+0000 7f42914ff640 10 monclient: get_monmap_and_config 2024-03-25T19:39:44.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.329+0000 7f42914ff640 10 monclient: build_initial_monmap 2024-03-25T19:39:44.329 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.329+0000 7f42914ff640 10 monclient: monmap: 2024-03-25T19:39:44.329 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:44.329 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:44.329 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:39:44.330105+0000 2024-03-25T19:39:44.329 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:39:44.330105+0000 2024-03-25T19:39:44.330 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:44.330 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:44.330 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:44.330 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:39:44.330 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:39:44.330 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:44.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.329+0000 7f42914ff640 1 Processor -- start 2024-03-25T19:39:44.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.329+0000 7f42914ff640 1 -- start start 2024-03-25T19:39:44.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.329+0000 7f42914ff640 10 monclient: init 2024-03-25T19:39:44.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:44.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:39:44.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 10 monclient(hunting): picked mon.noname-a con 0x7f428c13f990 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:44.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 10 monclient(hunting): picked mon.noname-c con 0x7f428c13fd70 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:44.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 10 monclient(hunting): picked mon.noname-b con 0x7f428c0afa60 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:44.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428c0afeb0 con 0x7f428c13f990 2024-03-25T19:39:44.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428c0b0100 con 0x7f428c13fd70 2024-03-25T19:39:44.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428c0b1150 con 0x7f428c0afa60 2024-03-25T19:39:44.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:44.332 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.330+0000 7f42914ff640 10 monclient(hunting): authenticate will time out at 6600.674301s 2024-03-25T19:39:44.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.337+0000 7f428a7fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f428c13fd70 0x7f428c145850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:58822/0 (socket says 172.21.15.143:58822) 2024-03-25T19:39:44.336 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.337+0000 7f428a7fc640 1 -- 172.21.15.143:0/2828025447 learned_addr learned my addr 172.21.15.143:0/2828025447 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:44.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.356+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931729621 0 0) 0x7f428c0b0100 con 0x7f428c13fd70 2024-03-25T19:39:44.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.356+0000 7f4289ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:44.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.356+0000 7f4289ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:44.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.356+0000 7f4289ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:44.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.356+0000 7f4289ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:39:44.356 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.356+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4274003520 con 0x7f428c13fd70 2024-03-25T19:39:44.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.362+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884760043 0 0) 0x7f428c0afeb0 con 0x7f428c13f990 2024-03-25T19:39:44.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.362+0000 7f4289ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:39:44.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.362+0000 7f4289ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:44.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.362+0000 7f4289ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:44.362 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.362+0000 7f4289ffb640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:39:44.363 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.362+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f428c0b0100 con 0x7f428c13f990 2024-03-25T19:39:44.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910303010 0 0) 0x7f4274003520 con 0x7f428c13fd70 2024-03-25T19:39:44.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 >> v1:172.21.15.143:6790/0 conn(0x7f428c0afa60 legacy=0x7f428c14dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:44.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 >> v1:172.21.15.143:6789/0 conn(0x7f428c13f990 legacy=0x7f428c06ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:44.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:39:44.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 1 monclient: found mon.noname-c 2024-03-25T19:39:44.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c0aede0 con 0x7f428c13fd70 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 10 monclient: _finish_auth 0 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4280001690 con 0x7f428c13fd70 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f42914ff640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f4289ffb640 10 monclient: dump: 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.369 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:44.370 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:44.370 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:44.370 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:44.370 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:44.370 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:44.370 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.368+0000 7f42914ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:44.392 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.393+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42800017f0 con 0x7f428c13fd70 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.393+0000 7f4289ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.393+0000 7f4289ffb640 1 -- 172.21.15.143:0/2828025447 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4280001c00 con 0x7f428c13fd70 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.393+0000 7f42914ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.393+0000 7f42914ff640 4 set_mon_vals no callback set 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.393+0000 7f4289ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.393+0000 7f4289ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.393+0000 7f4289ffb640 10 monclient: dump: 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:44.393 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:44.394 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:44.394 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:44.394 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:44.394 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:44.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 10 monclient: shutdown 2024-03-25T19:39:44.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:44.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 1 -- 172.21.15.143:0/2828025447 >> v1:172.21.15.203:6789/0 conn(0x7f428c13fd70 legacy=0x7f428c145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:44.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 1 -- 172.21.15.143:0/2828025447 shutdown_connections 2024-03-25T19:39:44.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 1 -- 172.21.15.143:0/2828025447 >> 172.21.15.143:0/2828025447 conn(0x7f428c0b4b80 msgr2=0x7f428c0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:44.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 1 -- 172.21.15.143:0/2828025447 shutdown_connections 2024-03-25T19:39:44.395 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 1 -- 172.21.15.143:0/2828025447 wait complete. 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 10 monclient: build_initial_monmap 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.394+0000 7f42914ff640 10 monclient: monmap: 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.395+0000 7f42914ff640 1 librados: starting msgr at 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.395+0000 7f42914ff640 1 librados: starting objecter 2024-03-25T19:39:44.396 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.395+0000 7f42914ff640 1 Processor -- start 2024-03-25T19:39:44.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.395+0000 7f42914ff640 1 -- start start 2024-03-25T19:39:44.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.395+0000 7f42914ff640 1 librados: setting wanted keys 2024-03-25T19:39:44.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.395+0000 7f42914ff640 1 librados: calling monclient init 2024-03-25T19:39:44.397 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.395+0000 7f42914ff640 10 monclient: init 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 10 monclient(hunting): picked mon.noname-a con 0x7f428c0afa60 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 10 monclient(hunting): picked mon.noname-c con 0x7f428c13f990 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 10 monclient(hunting): picked mon.noname-b con 0x7f428c13fd70 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428c11f250 con 0x7f428c0afa60 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428c11f4c0 con 0x7f428c13fd70 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428c11d2b0 con 0x7f428c13f990 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:44.398 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.396+0000 7f42914ff640 10 monclient(hunting): authenticate will time out at 6600.740313s 2024-03-25T19:39:44.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.401+0000 7f428b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f428c13fd70 0x7f428c1e2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:58884/0 (socket says 172.21.15.143:58884) 2024-03-25T19:39:44.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.401+0000 7f428b7fe640 1 -- 172.21.15.143:0/2171847478 learned_addr learned my addr 172.21.15.143:0/2171847478 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:44.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.402+0000 7f42737fe640 10 client.?.objecter ms_handle_connect 0x7f428c13fd70 2024-03-25T19:39:44.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.402+0000 7f42737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:44.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.410+0000 7f42737fe640 10 client.?.objecter ms_handle_connect 0x7f428c0afa60 2024-03-25T19:39:44.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.410+0000 7f42737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:44.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.413+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235238681 0 0) 0x7f428c11f4c0 con 0x7f428c13fd70 2024-03-25T19:39:44.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.413+0000 7f42737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:44.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.413+0000 7f42737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:44.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.413+0000 7f42737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:44.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.413+0000 7f42737fe640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:39:44.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.413+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4278004eb0 con 0x7f428c13fd70 2024-03-25T19:39:44.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.419+0000 7f42737fe640 10 client.?.objecter ms_handle_connect 0x7f428c13f990 2024-03-25T19:39:44.418 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.419+0000 7f42737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:44.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.421+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025944893 0 0) 0x7f428c11f250 con 0x7f428c0afa60 2024-03-25T19:39:44.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.421+0000 7f42737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:44.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.421+0000 7f42737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:44.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.421+0000 7f42737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:44.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.421+0000 7f42737fe640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:39:44.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.421+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f428c11f4c0 con 0x7f428c0afa60 2024-03-25T19:39:44.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.432+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 556500602 0 0) 0x7f4278004eb0 con 0x7f428c13fd70 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f428c11f250 con 0x7f428c13fd70 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4284001cd0 con 0x7f428c13fd70 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 10 monclient(hunting): dump: 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:44.433 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:44.434 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:44.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:44.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472422408 0 0) 0x7f428c11d2b0 con 0x7f428c13f990 2024-03-25T19:39:44.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:39:44.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:44.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:44.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:39:44.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.433+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4278004eb0 con 0x7f428c13f990 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.434+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448306215 0 0) 0x7f428c11f4c0 con 0x7f428c0afa60 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.434+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f428c11d2b0 con 0x7f428c0afa60 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.434+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f427c001930 con 0x7f428c0afa60 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.434+0000 7f42737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.434+0000 7f42737fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.434+0000 7f42737fe640 10 monclient(hunting): dump: 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.435 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:44.436 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:44.436 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:44.436 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:44.436 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:44.436 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:44.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 620883109 0 0) 0x7f428c11f250 con 0x7f428c13fd70 2024-03-25T19:39:44.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 >> v1:172.21.15.143:6790/0 conn(0x7f428c13f990 legacy=0x7f428c11d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:44.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 >> v1:172.21.15.143:6789/0 conn(0x7f428c0afa60 legacy=0x7f428c13bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:44.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 10 monclient: _finish_hunting 0 2024-03-25T19:39:44.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 1 monclient: found mon.b 2024-03-25T19:39:44.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:44.445 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428c11ce60 con 0x7f428c13fd70 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 10 monclient: _finish_auth 0 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 10 monclient: _check_auth_tickets 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 5 monclient: authenticate success, global_id 4672 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 10 monclient: _renew_subs 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428c1db790 con 0x7f428c13fd70 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 10 monclient: _renew_subs 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f428c1daf00 con 0x7f428c13fd70 2024-03-25T19:39:44.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.445+0000 7f42914ff640 1 librados: init done 2024-03-25T19:39:44.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.446+0000 7f42914ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:44.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.446+0000 7f42914ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:44.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.446+0000 7f42914ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:44.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.446+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f428c11d440 con 0x7f428c13fd70 2024-03-25T19:39:44.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.738+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4284003990 con 0x7f428c13fd70 2024-03-25T19:39:44.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.738+0000 7f42737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:44.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.738+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f4284003d20 con 0x7f428c13fd70 2024-03-25T19:39:44.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.738+0000 7f42737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:44.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.738+0000 7f42737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:44.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.738+0000 7f42737fe640 10 monclient: dump: 2024-03-25T19:39:44.738 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:44.738 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:44.739 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.738+0000 7f42897fa640 4 set_mon_vals no callback set 2024-03-25T19:39:44.740 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.739+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f4284003f40 con 0x7f428c13fd70 2024-03-25T19:39:44.740 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.740+0000 7f428affd640 10 monclient: get_auth_request con 0x7f427806dfb0 auth_method 0 2024-03-25T19:39:44.740 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.740+0000 7f42737fe640 10 client.4672.objecter ms_handle_connect 0x7f427806dfb0 2024-03-25T19:39:44.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.744+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f4284054aa0 con 0x7f428c13fd70 2024-03-25T19:39:44.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.744+0000 7f42737fe640 10 client.4672.objecter ms_dispatch 0x7f428c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:44.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.744+0000 7f42737fe640 3 client.4672.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:44.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.744+0000 7f42737fe640 3 client.4672.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:44.744 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.744+0000 7f42737fe640 20 client.4672.objecter dump_active .. 0 homeless 2024-03-25T19:39:44.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.745+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42840505b0 con 0x7f428c13fd70 2024-03-25T19:39:44.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.745+0000 7f42737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:44.745 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.745+0000 7f42737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:44.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.970+0000 7f42914ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:44.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.970+0000 7f42914ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:44.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.970+0000 7f42914ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:44.970 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.970+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f428c06bac0 con 0x7f428c13fd70 2024-03-25T19:39:44.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.998+0000 7f42737fe640 1 -- 172.21.15.143:0/2171847478 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f42840524e0 con 0x7f428c13fd70 2024-03-25T19:39:44.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.998+0000 7f42737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:44.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:44.998+0000 7f42737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:44.999 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:39:44.999 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:45.001 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 10 librados: watch_flush enter 2024-03-25T19:39:45.001 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 10 librados: watch_flush exit 2024-03-25T19:39:45.001 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 20 client.4672.objecter shutdown clearing up homeless session... 2024-03-25T19:39:45.001 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 10 client.4672.objecter successfully canceled tick 2024-03-25T19:39:45.001 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 >> v1:172.21.15.143:6800/1292282666 conn(0x7f427806dfb0 legacy=0x7f4278070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:45.001 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 10 monclient: shutdown 2024-03-25T19:39:45.001 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 >> v1:172.21.15.203:6789/0 conn(0x7f428c13fd70 legacy=0x7f428c1e2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 shutdown_connections 2024-03-25T19:39:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 >> 172.21.15.143:0/2171847478 conn(0x7f428c0b4b80 msgr2=0x7f428c06f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 shutdown_connections 2024-03-25T19:39:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 1 -- 172.21.15.143:0/2171847478 wait complete. 2024-03-25T19:39:45.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:45.001+0000 7f42914ff640 1 librados: shutdown 2024-03-25T19:39:45.014 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:51.820 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:51.939 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.939+0000 7f9897b0e640 10 monclient: get_monmap_and_config 2024-03-25T19:39:51.939 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.939+0000 7f9897b0e640 10 monclient: build_initial_monmap 2024-03-25T19:39:51.939 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.939+0000 7f9897b0e640 10 monclient: monmap: 2024-03-25T19:39:51.939 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:51.939 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:51.939 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:39:51.940672+0000 2024-03-25T19:39:51.940 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:39:51.940672+0000 2024-03-25T19:39:51.940 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:51.940 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:51.940 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:51.940 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:39:51.940 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:39:51.940 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:51.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.940+0000 7f9897b0e640 1 Processor -- start 2024-03-25T19:39:51.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.940+0000 7f9897b0e640 1 -- start start 2024-03-25T19:39:51.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.940+0000 7f9897b0e640 10 monclient: init 2024-03-25T19:39:51.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:51.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:39:51.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f989013fc60 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:51.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f9890070570 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:51.941 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f98900727a0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:51.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989013c9f0 con 0x7f989013fc60 2024-03-25T19:39:51.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989006fac0 con 0x7f98900727a0 2024-03-25T19:39:51.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98900715d0 con 0x7f9890070570 2024-03-25T19:39:51.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:51.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.941+0000 7f9897b0e640 10 monclient(hunting): authenticate will time out at 6608.285359s 2024-03-25T19:39:51.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.954+0000 7f98950a9640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f9890070570 0x7f9890072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:51894/0 (socket says 172.21.15.143:51894) 2024-03-25T19:39:51.954 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.954+0000 7f98950a9640 1 -- 172.21.15.143:0/917141356 learned_addr learned my addr 172.21.15.143:0/917141356 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:51.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.969+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686029414 0 0) 0x7f989006fac0 con 0x7f98900727a0 2024-03-25T19:39:51.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.969+0000 7f98948a8640 10 monclient(hunting): _negotiate 2024-03-25T19:39:51.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.969+0000 7f98948a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:51.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.969+0000 7f98948a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:51.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.969+0000 7f98948a8640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:39:51.969 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.969+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f987c003520 con 0x7f98900727a0 2024-03-25T19:39:51.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.971+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387750396 0 0) 0x7f989013c9f0 con 0x7f989013fc60 2024-03-25T19:39:51.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.971+0000 7f98948a8640 10 monclient(hunting): _negotiate 2024-03-25T19:39:51.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.971+0000 7f98948a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:51.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.971+0000 7f98948a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:51.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.971+0000 7f98948a8640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:39:51.971 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.971+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f989006fac0 con 0x7f989013fc60 2024-03-25T19:39:51.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.976+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601306626 0 0) 0x7f98900715d0 con 0x7f9890070570 2024-03-25T19:39:51.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.976+0000 7f98948a8640 10 monclient(hunting): _negotiate 2024-03-25T19:39:51.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.976+0000 7f98948a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:51.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.976+0000 7f98948a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:51.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.976+0000 7f98948a8640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:39:51.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.976+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f989013c9f0 con 0x7f9890070570 2024-03-25T19:39:51.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975903622 0 0) 0x7f987c003520 con 0x7f98900727a0 2024-03-25T19:39:51.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 >> v1:172.21.15.143:6790/0 conn(0x7f9890070570 legacy=0x7f9890072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:51.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 >> v1:172.21.15.143:6789/0 conn(0x7f989013fc60 legacy=0x7f989006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:51.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 10 monclient: _finish_hunting 0 2024-03-25T19:39:51.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 1 monclient: found mon.noname-c 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900718b0 con 0x7f98900727a0 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 10 monclient: _finish_auth 0 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 10 monclient: _check_auth_tickets 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9888001a10 con 0x7f98900727a0 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f9897b0e640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f9897b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f98948a8640 10 monclient: dump: 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:51.993 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:51.994 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:51.994 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:51.994 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:51.994 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:51.994 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:51.994 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:51.994 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:51.994 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:51.992+0000 7f9897b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.017+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9888001b70 con 0x7f98900727a0 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.017+0000 7f98948a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.017+0000 7f98948a8640 1 -- 172.21.15.143:0/917141356 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9888001f80 con 0x7f98900727a0 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.017+0000 7f9897b0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.017+0000 7f9897b0e640 4 set_mon_vals no callback set 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.017+0000 7f98948a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.017+0000 7f98948a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.017+0000 7f98948a8640 10 monclient: dump: 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:52.017 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.018 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.018 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:52.018 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:52.018 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:52.018 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:52.018 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:52.018 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:52.020 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.018+0000 7f9897b0e640 10 monclient: shutdown 2024-03-25T19:39:52.020 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.018+0000 7f9897b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.018+0000 7f9897b0e640 1 -- 172.21.15.143:0/917141356 >> v1:172.21.15.203:6789/0 conn(0x7f98900727a0 legacy=0x7f989014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.018+0000 7f9897b0e640 1 -- 172.21.15.143:0/917141356 shutdown_connections 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.018+0000 7f9897b0e640 1 -- 172.21.15.143:0/917141356 >> 172.21.15.143:0/917141356 conn(0x7f98900b4a80 msgr2=0x7f98900b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.018+0000 7f9897b0e640 1 -- 172.21.15.143:0/917141356 shutdown_connections 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.018+0000 7f9897b0e640 1 -- 172.21.15.143:0/917141356 wait complete. 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.019+0000 7f9897b0e640 10 monclient: build_initial_monmap 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.019+0000 7f9897b0e640 10 monclient: monmap: 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:39:52.021 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.019+0000 7f9897b0e640 1 librados: starting msgr at 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.019+0000 7f9897b0e640 1 librados: starting objecter 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.019+0000 7f9897b0e640 1 Processor -- start 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 1 -- start start 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 1 librados: setting wanted keys 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 1 librados: calling monclient init 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 10 monclient: init 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f9890070570 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f98900727a0 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:52.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f989013fc60 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:52.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98901d7a60 con 0x7f98900727a0 2024-03-25T19:39:52.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.020+0000 7f9897b0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98901d8630 con 0x7f9890070570 2024-03-25T19:39:52.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.021+0000 7f9897b0e640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98901d87c0 con 0x7f989013fc60 2024-03-25T19:39:52.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.021+0000 7f9897b0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:52.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.021+0000 7f9897b0e640 10 monclient(hunting): authenticate will time out at 6608.364677s 2024-03-25T19:39:52.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.035+0000 7f98958aa640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9890070570 0x7f9890142ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:51088/0 (socket says 172.21.15.143:51088) 2024-03-25T19:39:52.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.035+0000 7f98958aa640 1 -- 172.21.15.143:0/473080263 learned_addr learned my addr 172.21.15.143:0/473080263 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:52.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.036+0000 7f98867fc640 10 client.?.objecter ms_handle_connect 0x7f9890070570 2024-03-25T19:39:52.036 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.036+0000 7f98867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:52.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.040+0000 7f98867fc640 10 client.?.objecter ms_handle_connect 0x7f98900727a0 2024-03-25T19:39:52.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.040+0000 7f98867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:52.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.047+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94441491 0 0) 0x7f98901d8630 con 0x7f9890070570 2024-03-25T19:39:52.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.047+0000 7f98867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:52.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.047+0000 7f98867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:52.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.047+0000 7f98867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:52.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.047+0000 7f98867fc640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:39:52.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.047+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9874004ee0 con 0x7f9890070570 2024-03-25T19:39:52.047 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.048+0000 7f98867fc640 10 client.?.objecter ms_handle_connect 0x7f989013fc60 2024-03-25T19:39:52.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.048+0000 7f98867fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:52.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.053+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55245088 0 0) 0x7f98901d7a60 con 0x7f98900727a0 2024-03-25T19:39:52.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.053+0000 7f98867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:52.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.053+0000 7f98867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:52.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.053+0000 7f98867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:52.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.053+0000 7f98867fc640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:39:52.053 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.053+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98901d8630 con 0x7f98900727a0 2024-03-25T19:39:52.057 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.057+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 972987524 0 0) 0x7f9874004ee0 con 0x7f9890070570 2024-03-25T19:39:52.057 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.057+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98901d7a60 con 0x7f9890070570 2024-03-25T19:39:52.057 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.057+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f988c001990 con 0x7f9890070570 2024-03-25T19:39:52.057 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.057+0000 7f98867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:52.057 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.057+0000 7f98867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:52.057 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.057+0000 7f98867fc640 10 monclient(hunting): dump: 2024-03-25T19:39:52.057 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:52.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.057+0000 7f98867fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:39:52.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.058+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193882042 0 0) 0x7f98901d87c0 con 0x7f989013fc60 2024-03-25T19:39:52.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.058+0000 7f98867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:52.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.058+0000 7f98867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:52.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.058+0000 7f98867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:52.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.058+0000 7f98867fc640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:39:52.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.058+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9874004ee0 con 0x7f989013fc60 2024-03-25T19:39:52.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.067+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1768612093 0 0) 0x7f98901d8630 con 0x7f98900727a0 2024-03-25T19:39:52.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.067+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98901d87c0 con 0x7f98900727a0 2024-03-25T19:39:52.066 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.067+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9880001870 con 0x7f98900727a0 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.067+0000 7f98867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.067+0000 7f98867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.067+0000 7f98867fc640 10 monclient(hunting): dump: 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:52.067 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:52.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 842241618 0 0) 0x7f9874004ee0 con 0x7f989013fc60 2024-03-25T19:39:52.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98901d8630 con 0x7f989013fc60 2024-03-25T19:39:52.079 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f9888001e80 con 0x7f989013fc60 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 10 monclient(hunting): dump: 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3984328884 0 0) 0x7f98901d7a60 con 0x7f9890070570 2024-03-25T19:39:52.080 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 >> v1:172.21.15.143:6790/0 conn(0x7f989013fc60 legacy=0x7f98901d7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 >> v1:172.21.15.143:6789/0 conn(0x7f98900727a0 legacy=0x7f98900ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 1 monclient: found mon.b 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989013d6b0 con 0x7f9890070570 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f98867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 10 monclient: _renew_subs 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989013d960 con 0x7f9890070570 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 10 client.4678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:39:52.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 10 monclient: _renew_subs 2024-03-25T19:39:52.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:52.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98901359a0 con 0x7f9890070570 2024-03-25T19:39:52.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.079+0000 7f9897b0e640 1 librados: init done 2024-03-25T19:39:52.082 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.080+0000 7f9897b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:52.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.080+0000 7f9897b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:52.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.080+0000 7f9897b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:52.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.080+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98900627b0 con 0x7f9890070570 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.366+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f988c0035f0 con 0x7f9890070570 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.366+0000 7f98867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.366+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f988c0037e0 con 0x7f9890070570 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.366+0000 7f98867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.366+0000 7f98867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.366+0000 7f98867fc640 10 monclient: dump: 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:52.366 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.366+0000 7f9887fff640 4 set_mon_vals no callback set 2024-03-25T19:39:52.367 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.367+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f988c003a00 con 0x7f9890070570 2024-03-25T19:39:52.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.368+0000 7f98950a9640 10 monclient: get_auth_request con 0x7f987406e190 auth_method 0 2024-03-25T19:39:52.368 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.368+0000 7f98867fc640 10 client.4678.objecter ms_handle_connect 0x7f987406e190 2024-03-25T19:39:52.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.372+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3854672934 0 0) 0x7f988c0545e0 con 0x7f9890070570 2024-03-25T19:39:52.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.372+0000 7f98867fc640 10 client.4678.objecter ms_dispatch 0x7f9890000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:39:52.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.372+0000 7f98867fc640 3 client.4678.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:39:52.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.372+0000 7f98867fc640 3 client.4678.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:39:52.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.372+0000 7f98867fc640 20 client.4678.objecter dump_active .. 0 homeless 2024-03-25T19:39:52.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.372+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f988c0500f0 con 0x7f9890070570 2024-03-25T19:39:52.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.372+0000 7f98867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:39:52.372 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.372+0000 7f98867fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:39:52.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.623+0000 7f9897b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:52.623 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.623+0000 7f9897b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:52.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.623+0000 7f9897b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:39:52.624 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.623+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98900b2d60 con 0x7f9890070570 2024-03-25T19:39:52.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.778+0000 7f98867fc640 1 -- 172.21.15.143:0/473080263 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 935802113) 0x7f988c052020 con 0x7f9890070570 2024-03-25T19:39:52.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.778+0000 7f98867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:39:52.778 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.778+0000 7f98867fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:39:52.778 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:39:52.778 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":10,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:30:01.479963+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:01.479963+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:39:52.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 10 librados: watch_flush enter 2024-03-25T19:39:52.780 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 10 librados: watch_flush exit 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 20 client.4678.objecter shutdown clearing up homeless session... 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 10 client.4678.objecter successfully canceled tick 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 >> v1:172.21.15.143:6800/1292282666 conn(0x7f987406e190 legacy=0x7f9874070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 10 monclient: shutdown 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 >> v1:172.21.15.203:6789/0 conn(0x7f9890070570 legacy=0x7f9890142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 shutdown_connections 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.780+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 >> 172.21.15.143:0/473080263 conn(0x7f98900b4a80 msgr2=0x7f98900afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.781+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 shutdown_connections 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.781+0000 7f9897b0e640 1 -- 172.21.15.143:0/473080263 wait complete. 2024-03-25T19:39:52.781 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:52.781+0000 7f9897b0e640 1 librados: shutdown 2024-03-25T19:39:52.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:39:59.597 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.706+0000 7fdf61d7c640 10 monclient: get_monmap_and_config 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.706+0000 7fdf61d7c640 10 monclient: build_initial_monmap 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.706+0000 7fdf61d7c640 10 monclient: monmap: 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:39:59.707754+0000 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:39:59.707754+0000 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:59.707 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:59.708 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:39:59.708 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:39:59.708 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:59.709 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.707+0000 7fdf61d7c640 1 Processor -- start 2024-03-25T19:39:59.709 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.707+0000 7fdf61d7c640 1 -- start start 2024-03-25T19:39:59.709 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.707+0000 7fdf61d7c640 10 monclient: init 2024-03-25T19:39:59.709 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:59.709 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:39:59.709 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 10 monclient(hunting): picked mon.noname-c con 0x7fdf5c0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:59.709 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 10 monclient(hunting): picked mon.noname-a con 0x7fdf5c13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:59.709 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 10 monclient(hunting): picked mon.noname-b con 0x7fdf5c1430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:59.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c142520 con 0x7fdf5c13f710 2024-03-25T19:39:59.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c0b28a0 con 0x7fdf5c0a98f0 2024-03-25T19:39:59.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c0b2a30 con 0x7fdf5c1430c0 2024-03-25T19:39:59.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:59.710 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.708+0000 7fdf61d7c640 10 monclient(hunting): authenticate will time out at 6616.052411s 2024-03-25T19:39:59.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.709+0000 7fdf5b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fdf5c0a98f0 0x7fdf5c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:51172/0 (socket says 172.21.15.143:51172) 2024-03-25T19:39:59.711 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.709+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/3188872939 learned_addr learned my addr 172.21.15.143:0/3188872939 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:59.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.799+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276189400 0 0) 0x7fdf5c142520 con 0x7fdf5c13f710 2024-03-25T19:39:59.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.799+0000 7fdf5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:59.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.799+0000 7fdf5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:59.799 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.799+0000 7fdf5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:59.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.799+0000 7fdf5a7fc640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:39:59.800 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.799+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf44003520 con 0x7fdf5c13f710 2024-03-25T19:39:59.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.802+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313021947 0 0) 0x7fdf5c0b2a30 con 0x7fdf5c1430c0 2024-03-25T19:39:59.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.802+0000 7fdf5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:39:59.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.802+0000 7fdf5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:59.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.802+0000 7fdf5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:59.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.802+0000 7fdf5a7fc640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:39:59.802 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.802+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c142520 con 0x7fdf5c1430c0 2024-03-25T19:39:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1113081811 0 0) 0x7fdf44003520 con 0x7fdf5c13f710 2024-03-25T19:39:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 >> v1:172.21.15.143:6790/0 conn(0x7fdf5c1430c0 legacy=0x7fdf5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:59.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 >> v1:172.21.15.203:6789/0 conn(0x7fdf5c0a98f0 legacy=0x7fdf5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0b44c0 con 0x7fdf5c13f710 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf61d7c640 5 monclient: authenticate success, global_id 4674 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf50001690 con 0x7fdf5c13f710 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.855+0000 7fdf5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.856+0000 7fdf5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.856+0000 7fdf5a7fc640 10 monclient: dump: 2024-03-25T19:39:59.856 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:59.857 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.856+0000 7fdf61d7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:39:59.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.861+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf500017f0 con 0x7fdf5c13f710 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.861+0000 7fdf5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.861+0000 7fdf5a7fc640 1 -- 172.21.15.143:0/3188872939 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf50001c00 con 0x7fdf5c13f710 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.861+0000 7fdf61d7c640 10 monclient: get_monmap_and_config success 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.861+0000 7fdf61d7c640 4 set_mon_vals no callback set 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.861+0000 7fdf5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.861+0000 7fdf5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.861+0000 7fdf5a7fc640 10 monclient: dump: 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:59.861 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:59.862 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:59.862 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:59.862 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:59.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.862+0000 7fdf61d7c640 10 monclient: shutdown 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.862+0000 7fdf61d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.862+0000 7fdf61d7c640 1 -- 172.21.15.143:0/3188872939 >> v1:172.21.15.143:6789/0 conn(0x7fdf5c13f710 legacy=0x7fdf5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.862+0000 7fdf61d7c640 1 -- 172.21.15.143:0/3188872939 shutdown_connections 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.862+0000 7fdf61d7c640 1 -- 172.21.15.143:0/3188872939 >> 172.21.15.143:0/3188872939 conn(0x7fdf5c0b7ea0 msgr2=0x7fdf5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.862+0000 7fdf61d7c640 1 -- 172.21.15.143:0/3188872939 shutdown_connections 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.862+0000 7fdf61d7c640 1 -- 172.21.15.143:0/3188872939 wait complete. 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 10 monclient: build_initial_monmap 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 10 monclient: monmap: 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:39:59.864 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 1 librados: starting msgr at 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 1 librados: starting objecter 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 1 Processor -- start 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 1 -- start start 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 1 librados: setting wanted keys 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 1 librados: calling monclient init 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.863+0000 7fdf61d7c640 10 monclient: init 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 10 monclient: _reopen_session rank -1 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:39:59.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 10 monclient(hunting): picked mon.noname-c con 0x7fdf5c0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:39:59.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 10 monclient(hunting): picked mon.noname-b con 0x7fdf5c13f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:39:59.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 10 monclient(hunting): picked mon.noname-a con 0x7fdf5c1430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:39:59.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c0abac0 con 0x7fdf5c1430c0 2024-03-25T19:39:59.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c0abda0 con 0x7fdf5c13f710 2024-03-25T19:39:59.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c1380e0 con 0x7fdf5c0a98f0 2024-03-25T19:39:59.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 10 monclient(hunting): _renew_subs 2024-03-25T19:39:59.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf61d7c640 10 monclient(hunting): authenticate will time out at 6616.207888s 2024-03-25T19:39:59.866 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf5b7fe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7fdf5c0a98f0 0x7fdf5c0ac580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:49688/0 (socket says 172.21.15.143:49688) 2024-03-25T19:39:59.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.864+0000 7fdf5b7fe640 1 -- 172.21.15.143:0/4248765486 learned_addr learned my addr 172.21.15.143:0/4248765486 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:39:59.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.865+0000 7fdf43fff640 10 client.?.objecter ms_handle_connect 0x7fdf5c0a98f0 2024-03-25T19:39:59.867 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.865+0000 7fdf43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:59.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.870+0000 7fdf43fff640 10 client.?.objecter ms_handle_connect 0x7fdf5c13f710 2024-03-25T19:39:59.870 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.870+0000 7fdf43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:59.880 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:26.445 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.927+0000 7fdf43fff640 10 client.?.objecter ms_handle_connect 0x7fdf5c1430c0 2024-03-25T19:39:59.927 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.927+0000 7fdf43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:39:59.942 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:26.507 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.948+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996527101 0 0) 0x7fdf5c0abac0 con 0x7fdf5c1430c0 2024-03-25T19:39:59.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.948+0000 7fdf43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:59.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.948+0000 7fdf43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:59.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.948+0000 7fdf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:59.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.948+0000 7fdf43fff640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:39:59.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.948+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf48004eb0 con 0x7fdf5c1430c0 2024-03-25T19:39:59.967 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:26.532 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.985+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990918603 0 0) 0x7fdf5c0abda0 con 0x7fdf5c13f710 2024-03-25T19:39:59.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.985+0000 7fdf43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:59.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.985+0000 7fdf43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:59.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.985+0000 7fdf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:59.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.985+0000 7fdf43fff640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:39:59.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.985+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c0abac0 con 0x7fdf5c13f710 2024-03-25T19:39:59.989 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:26.525 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.995 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:26.560 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:39:59.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.996+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405607626 0 0) 0x7fdf5c1380e0 con 0x7fdf5c0a98f0 2024-03-25T19:39:59.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.997+0000 7fdf43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:39:59.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.997+0000 7fdf43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:39:59.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.997+0000 7fdf43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:39:59.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.997+0000 7fdf43fff640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:39:59.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.997+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c0abda0 con 0x7fdf5c0a98f0 2024-03-25T19:39:59.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.999+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2744151070 0 0) 0x7fdf5c0abac0 con 0x7fdf5c13f710 2024-03-25T19:39:59.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.999+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf5c1380e0 con 0x7fdf5c13f710 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.999+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf50001b00 con 0x7fdf5c13f710 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.999+0000 7fdf43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.999+0000 7fdf43fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.999+0000 7fdf43fff640 10 monclient(hunting): dump: 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:39:59.999 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:00.000 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:39:59.999+0000 7fdf43fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:00.000 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:26.563 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:00.003 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:26.538 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:00.034 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:26.569 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:00.035 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:26.600 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:00.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.041+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2508373136 0 0) 0x7fdf5c0abda0 con 0x7fdf5c0a98f0 2024-03-25T19:40:00.040 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.041+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf5c0abac0 con 0x7fdf5c0a98f0 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.041+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf4c0018b0 con 0x7fdf5c0a98f0 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.041+0000 7fdf43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.041+0000 7fdf43fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.041+0000 7fdf43fff640 10 monclient(hunting): dump: 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:00.041 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:00.042 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:00.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.091+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1646134282 0 0) 0x7fdf5c1380e0 con 0x7fdf5c13f710 2024-03-25T19:40:00.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 >> v1:172.21.15.143:6790/0 conn(0x7fdf5c0a98f0 legacy=0x7fdf5c0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 >> v1:172.21.15.143:6789/0 conn(0x7fdf5c1430c0 legacy=0x7fdf5c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.092 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 1 monclient: found mon.b 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c137d90 con 0x7fdf5c13f710 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 10 monclient: _finish_auth 0 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 5 monclient: authenticate success, global_id 4684 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 10 monclient: _renew_subs 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c139240 con 0x7fdf5c13f710 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 10 client.4684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 10 monclient: _renew_subs 2024-03-25T19:40:00.093 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:00.094 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c136830 con 0x7fdf5c13f710 2024-03-25T19:40:00.094 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.092+0000 7fdf61d7c640 1 librados: init done 2024-03-25T19:40:00.094 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.093+0000 7fdf61d7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:00.094 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.093+0000 7fdf61d7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:00.094 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.093+0000 7fdf61d7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:00.094 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.093+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c0627b0 con 0x7fdf5c13f710 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.420+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf500038a0 con 0x7fdf5c13f710 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.420+0000 7fdf43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.420+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fdf50001250 con 0x7fdf5c13f710 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.420+0000 7fdf43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.420+0000 7fdf43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.420+0000 7fdf43fff640 10 monclient: dump: 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:00.420 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:00.421 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:00.421 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:00.421 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:00.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.420+0000 7fdf59ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:00.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.421+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fdf50019a20 con 0x7fdf5c13f710 2024-03-25T19:40:00.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.421+0000 7fdf5b7fe640 10 monclient: get_auth_request con 0x7fdf4806df80 auth_method 0 2024-03-25T19:40:00.421 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.421+0000 7fdf43fff640 10 client.4684.objecter ms_handle_connect 0x7fdf4806df80 2024-03-25T19:40:00.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.426+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7fdf50053660 con 0x7fdf5c13f710 2024-03-25T19:40:00.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.426+0000 7fdf43fff640 10 client.4684.objecter ms_dispatch 0x7fdf5c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:00.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.426+0000 7fdf43fff640 3 client.4684.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:00.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.426+0000 7fdf43fff640 3 client.4684.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:00.426 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.426+0000 7fdf43fff640 20 client.4684.objecter dump_active .. 0 homeless 2024-03-25T19:40:00.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.427+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf5004efa0 con 0x7fdf5c13f710 2024-03-25T19:40:00.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.427+0000 7fdf43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:00.427 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.427+0000 7fdf43fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:00.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.673+0000 7fdf61d7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:00.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.673+0000 7fdf61d7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:00.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.673+0000 7fdf61d7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:00.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.673+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf5c13f070 con 0x7fdf5c13f710 2024-03-25T19:40:00.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.699+0000 7fdf43fff640 1 -- 172.21.15.143:0/4248765486 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7fdf500510a0 con 0x7fdf5c13f710 2024-03-25T19:40:00.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.699+0000 7fdf43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:00.699 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.699+0000 7fdf43fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:00.699 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:40:00.699 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:00.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.701+0000 7fdf61d7c640 10 librados: watch_flush enter 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.701+0000 7fdf61d7c640 10 librados: watch_flush exit 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.701+0000 7fdf61d7c640 20 client.4684.objecter shutdown clearing up homeless session... 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.701+0000 7fdf61d7c640 10 client.4684.objecter successfully canceled tick 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 >> v1:172.21.15.143:6800/1292282666 conn(0x7fdf4806df80 legacy=0x7fdf48070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 10 monclient: shutdown 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 >> v1:172.21.15.203:6789/0 conn(0x7fdf5c13f710 legacy=0x7fdf5c138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 shutdown_connections 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 >> 172.21.15.143:0/4248765486 conn(0x7fdf5c0b7ea0 msgr2=0x7fdf5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 shutdown_connections 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 1 -- 172.21.15.143:0/4248765486 wait complete. 2024-03-25T19:40:00.702 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:00.702+0000 7fdf61d7c640 1 librados: shutdown 2024-03-25T19:40:00.712 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:07.519 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:07.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.624+0000 7ff8feae6640 10 monclient: get_monmap_and_config 2024-03-25T19:40:07.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.624+0000 7ff8feae6640 10 monclient: build_initial_monmap 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.624+0000 7ff8feae6640 10 monclient: monmap: 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:40:07.626856+0000 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:40:07.626856+0000 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:40:07.626 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:07.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.625+0000 7ff8feae6640 1 Processor -- start 2024-03-25T19:40:07.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.625+0000 7ff8feae6640 1 -- start start 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.625+0000 7ff8feae6640 10 monclient: init 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8f80b4f10 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f80b44d0 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f813f740 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80b0140 con 0x7ff8f80b44d0 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80af3a0 con 0x7ff8f80b4f10 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80af530 con 0x7ff8f813f740 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:07.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.626+0000 7ff8feae6640 10 monclient(hunting): authenticate will time out at 6623.971208s 2024-03-25T19:40:07.635 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.634+0000 7ff8f7fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ff8f80b44d0 0x7ff8f80b16c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:33872/0 (socket says 172.21.15.143:33872) 2024-03-25T19:40:07.635 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.634+0000 7ff8f7fff640 1 -- 172.21.15.143:0/3725796144 learned_addr learned my addr 172.21.15.143:0/3725796144 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:07.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.645+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986622613 0 0) 0x7ff8f80b0140 con 0x7ff8f80b44d0 2024-03-25T19:40:07.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.645+0000 7ff8f77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:07.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.645+0000 7ff8f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:07.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.645+0000 7ff8f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:07.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.645+0000 7ff8f77fe640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:40:07.646 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.645+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e8003520 con 0x7ff8f80b44d0 2024-03-25T19:40:07.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.656+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502311321 0 0) 0x7ff8f80af3a0 con 0x7ff8f80b4f10 2024-03-25T19:40:07.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.656+0000 7ff8f77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:07.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.656+0000 7ff8f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:07.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.656+0000 7ff8f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:07.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.656+0000 7ff8f77fe640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:40:07.657 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.656+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80b0140 con 0x7ff8f80b4f10 2024-03-25T19:40:07.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3417574141 0 0) 0x7ff8e8003520 con 0x7ff8f80b44d0 2024-03-25T19:40:07.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 >> v1:172.21.15.143:6790/0 conn(0x7ff8f813f740 legacy=0x7ff8f813fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 >> v1:172.21.15.203:6789/0 conn(0x7ff8f80b4f10 legacy=0x7ff8f80ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 1 monclient: found mon.noname-a 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f80afcd0 con 0x7ff8f80b44d0 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff8e4001690 con 0x7ff8f80b44d0 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8feae6640 5 monclient: authenticate success, global_id 4680 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8feae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:07.660 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8f77fe640 10 monclient: dump: 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:07.661 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.658+0000 7ff8feae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.672+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8e40017f0 con 0x7ff8f80b44d0 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.672+0000 7ff8f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.672+0000 7ff8f77fe640 1 -- 172.21.15.143:0/3725796144 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff8e4001c00 con 0x7ff8f80b44d0 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.672+0000 7ff8feae6640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.672+0000 7ff8feae6640 4 set_mon_vals no callback set 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.672+0000 7ff8f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.672+0000 7ff8f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.672+0000 7ff8f77fe640 10 monclient: dump: 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:07.673 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:07.674 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.674 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.674 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:07.674 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:07.674 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:07.674 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:07.674 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:07.674 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:07.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.673+0000 7ff8feae6640 10 monclient: shutdown 2024-03-25T19:40:07.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.673+0000 7ff8feae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:07.676 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.673+0000 7ff8feae6640 1 -- 172.21.15.143:0/3725796144 >> v1:172.21.15.143:6789/0 conn(0x7ff8f80b44d0 legacy=0x7ff8f80b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.673+0000 7ff8feae6640 1 -- 172.21.15.143:0/3725796144 shutdown_connections 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.673+0000 7ff8feae6640 1 -- 172.21.15.143:0/3725796144 >> 172.21.15.143:0/3725796144 conn(0x7ff8f80617a0 msgr2=0x7ff8f8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.673+0000 7ff8feae6640 1 -- 172.21.15.143:0/3725796144 shutdown_connections 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.673+0000 7ff8feae6640 1 -- 172.21.15.143:0/3725796144 wait complete. 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 10 monclient: build_initial_monmap 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 10 monclient: monmap: 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:40:07.677 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 1 librados: starting msgr at 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 1 librados: starting objecter 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 1 Processor -- start 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 1 -- start start 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 1 librados: setting wanted keys 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 1 librados: calling monclient init 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.674+0000 7ff8feae6640 10 monclient: init 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.675+0000 7ff8feae6640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.675+0000 7ff8feae6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.675+0000 7ff8feae6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f80b44d0 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.675+0000 7ff8feae6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8f80b4f10 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:07.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.675+0000 7ff8feae6640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f813f740 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:07.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.675+0000 7ff8feae6640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80ae290 con 0x7ff8f813f740 2024-03-25T19:40:07.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.676+0000 7ff8feae6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80adf40 con 0x7ff8f80b44d0 2024-03-25T19:40:07.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.676+0000 7ff8feae6640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80abc30 con 0x7ff8f80b4f10 2024-03-25T19:40:07.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.676+0000 7ff8feae6640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:07.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.676+0000 7ff8feae6640 10 monclient(hunting): authenticate will time out at 6624.020921s 2024-03-25T19:40:07.681 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.681+0000 7ff8f7fff640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7ff8f80b4f10 0x7ff8f80aae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:39410/0 (socket says 172.21.15.143:39410) 2024-03-25T19:40:07.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.681+0000 7ff8f7fff640 1 -- 172.21.15.143:0/567105751 learned_addr learned my addr 172.21.15.143:0/567105751 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:07.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.682+0000 7ff8f57fa640 10 client.?.objecter ms_handle_connect 0x7ff8f80b4f10 2024-03-25T19:40:07.682 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.682+0000 7ff8f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:07.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.689+0000 7ff8f57fa640 10 client.?.objecter ms_handle_connect 0x7ff8f80b44d0 2024-03-25T19:40:07.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.689+0000 7ff8f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:07.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.690+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501693458 0 0) 0x7ff8f80abc30 con 0x7ff8f80b4f10 2024-03-25T19:40:07.690 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.690+0000 7ff8f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:07.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.690+0000 7ff8f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:07.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.690+0000 7ff8f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:07.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.690+0000 7ff8f57fa640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:40:07.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.690+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc004eb0 con 0x7ff8f80b4f10 2024-03-25T19:40:07.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.691+0000 7ff8f57fa640 10 client.?.objecter ms_handle_connect 0x7ff8f813f740 2024-03-25T19:40:07.691 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.691+0000 7ff8f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:07.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.700+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747135219 0 0) 0x7ff8f80adf40 con 0x7ff8f80b44d0 2024-03-25T19:40:07.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.700+0000 7ff8f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:07.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.700+0000 7ff8f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:07.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.700+0000 7ff8f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:07.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.700+0000 7ff8f57fa640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:40:07.701 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.700+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80abc30 con 0x7ff8f80b44d0 2024-03-25T19:40:07.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.705+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691601785 0 0) 0x7ff8f80ae290 con 0x7ff8f813f740 2024-03-25T19:40:07.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.705+0000 7ff8f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:07.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.705+0000 7ff8f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:07.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.705+0000 7ff8f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:07.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.705+0000 7ff8f57fa640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:40:07.706 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.705+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80adf40 con 0x7ff8f813f740 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.711+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1934359921 0 0) 0x7ff8dc004eb0 con 0x7ff8f80b4f10 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.711+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8f80ae290 con 0x7ff8f80b4f10 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.711+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff8e4001c40 con 0x7ff8f80b4f10 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.711+0000 7ff8f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.711+0000 7ff8f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.711+0000 7ff8f57fa640 10 monclient(hunting): dump: 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.712 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:07.713 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:07.713 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:07.713 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:07.713 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:07.713 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:07.713 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.711+0000 7ff8f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:07.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.713+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3657887330 0 0) 0x7ff8f80abc30 con 0x7ff8f80b44d0 2024-03-25T19:40:07.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.713+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8dc004eb0 con 0x7ff8f80b44d0 2024-03-25T19:40:07.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.713+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff8ec001970 con 0x7ff8f80b44d0 2024-03-25T19:40:07.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.713+0000 7ff8f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:07.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.714+0000 7ff8f57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:07.714 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.714+0000 7ff8f57fa640 10 monclient(hunting): dump: 2024-03-25T19:40:07.714 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:07.715 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.718+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54466076 0 0) 0x7ff8f80adf40 con 0x7ff8f813f740 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.718+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8f80abc30 con 0x7ff8f813f740 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.718+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff8f0001a60 con 0x7ff8f813f740 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.718+0000 7ff8f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.718+0000 7ff8f57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.718+0000 7ff8f57fa640 10 monclient(hunting): dump: 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:07.719 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:07.720 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:07.720 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:07.720 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:07.720 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:07.720 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:07.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1821364645 0 0) 0x7ff8f80ae290 con 0x7ff8f80b4f10 2024-03-25T19:40:07.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 >> v1:172.21.15.203:6789/0 conn(0x7ff8f80b44d0 legacy=0x7ff8f8131df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:07.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 >> v1:172.21.15.143:6789/0 conn(0x7ff8f813f740 legacy=0x7ff8f80ab520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:07.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:07.727 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 1 monclient: found mon.c 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f81316f0 con 0x7ff8f80b4f10 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 5 monclient: authenticate success, global_id 4676 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 10 monclient: _renew_subs 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f81319d0 con 0x7ff8f80b4f10 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 10 monclient: _renew_subs 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:40:07.728 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f80ad7c0 con 0x7ff8f80b4f10 2024-03-25T19:40:07.729 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.726+0000 7ff8feae6640 1 librados: init done 2024-03-25T19:40:07.729 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.727+0000 7ff8feae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:07.729 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.727+0000 7ff8feae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:07.729 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.727+0000 7ff8feae6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:40:07.729 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:07.727+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f80627b0 con 0x7ff8f80b4f10 2024-03-25T19:40:08.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.025+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8e4004a90 con 0x7ff8f80b4f10 2024-03-25T19:40:08.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.025+0000 7ff8f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:08.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.025+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ff8e4004de0 con 0x7ff8f80b4f10 2024-03-25T19:40:08.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.025+0000 7ff8f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:08.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.025+0000 7ff8f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:08.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.025+0000 7ff8f57fa640 10 monclient: dump: 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:08.027 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.025+0000 7ff8f6ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:08.028 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.026+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7ff8e4005000 con 0x7ff8f80b4f10 2024-03-25T19:40:08.028 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.026+0000 7ff8fc882640 10 monclient: get_auth_request con 0x7ff8dc06e0c0 auth_method 0 2024-03-25T19:40:08.028 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.027+0000 7ff8f57fa640 10 client.4676.objecter ms_handle_connect 0x7ff8dc06e0c0 2024-03-25T19:40:08.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.033+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7ff8e4054980 con 0x7ff8f80b4f10 2024-03-25T19:40:08.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.033+0000 7ff8f57fa640 10 client.4676.objecter ms_dispatch 0x7ff8f8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:08.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.033+0000 7ff8f57fa640 3 client.4676.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:08.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.033+0000 7ff8f57fa640 3 client.4676.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:08.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.033+0000 7ff8f57fa640 20 client.4676.objecter dump_active .. 0 homeless 2024-03-25T19:40:08.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.033+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8e4050490 con 0x7ff8f80b4f10 2024-03-25T19:40:08.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.033+0000 7ff8f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:08.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.033+0000 7ff8f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:08.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.291+0000 7ff8feae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:08.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.291+0000 7ff8feae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:08.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.291+0000 7ff8feae6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:40:08.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.291+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8f80b3160 con 0x7ff8f80b4f10 2024-03-25T19:40:08.321 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.321+0000 7ff8f57fa640 1 -- 172.21.15.143:0/567105751 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7ff8e40523c0 con 0x7ff8f80b4f10 2024-03-25T19:40:08.322 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.321+0000 7ff8f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:08.322 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.321+0000 7ff8f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:08.322 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:40:08.322 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:08.324 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.323+0000 7ff8feae6640 10 librados: watch_flush enter 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.323+0000 7ff8feae6640 10 librados: watch_flush exit 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.323+0000 7ff8feae6640 20 client.4676.objecter shutdown clearing up homeless session... 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.323+0000 7ff8feae6640 10 client.4676.objecter successfully canceled tick 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.323+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 >> v1:172.21.15.143:6800/1292282666 conn(0x7ff8dc06e0c0 legacy=0x7ff8dc070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.324+0000 7ff8feae6640 10 monclient: shutdown 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.324+0000 7ff8feae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.324+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 >> v1:172.21.15.143:6790/0 conn(0x7ff8f80b4f10 legacy=0x7ff8f80aae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.324+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 shutdown_connections 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.324+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 >> 172.21.15.143:0/567105751 conn(0x7ff8f80617a0 msgr2=0x7ff8f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.324+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 shutdown_connections 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.324+0000 7ff8feae6640 1 -- 172.21.15.143:0/567105751 wait complete. 2024-03-25T19:40:08.325 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:08.324+0000 7ff8feae6640 1 librados: shutdown 2024-03-25T19:40:08.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:15.143 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:15.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.256+0000 7f156c382640 10 monclient: get_monmap_and_config 2024-03-25T19:40:15.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.256+0000 7f156c382640 10 monclient: build_initial_monmap 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.257+0000 7f156c382640 10 monclient: monmap: 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:40:15.257904+0000 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:40:15.257904+0000 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:40:15.257 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:15.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.257+0000 7f156c382640 1 Processor -- start 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.257+0000 7f156c382640 1 -- start start 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.257+0000 7f156c382640 10 monclient: init 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.258+0000 7f156c382640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.258+0000 7f156c382640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.259+0000 7f156c382640 10 monclient(hunting): picked mon.noname-c con 0x7f15640a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.259+0000 7f156c382640 10 monclient(hunting): picked mon.noname-a con 0x7f156413f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.259+0000 7f156c382640 10 monclient(hunting): picked mon.noname-b con 0x7f15641430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.259+0000 7f156c382640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1564142520 con 0x7f156413f710 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.259+0000 7f156c382640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15640b28a0 con 0x7f15640a98f0 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.259+0000 7f156c382640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15640b2a30 con 0x7f15641430c0 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.259+0000 7f156c382640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:15.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.259+0000 7f156c382640 10 monclient(hunting): authenticate will time out at 6631.602776s 2024-03-25T19:40:15.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.269+0000 7f156a11e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f15640a98f0 0x7f15640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:34240/0 (socket says 172.21.15.143:34240) 2024-03-25T19:40:15.269 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.269+0000 7f156a11e640 1 -- 172.21.15.143:0/1792090861 learned_addr learned my addr 172.21.15.143:0/1792090861 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:15.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.280+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111805244 0 0) 0x7f15640b28a0 con 0x7f15640a98f0 2024-03-25T19:40:15.280 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.280+0000 7f156911c640 10 monclient(hunting): _negotiate 2024-03-25T19:40:15.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.280+0000 7f156911c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:15.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.280+0000 7f156911c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:15.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.280+0000 7f156911c640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:40:15.281 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.280+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1550003520 con 0x7f15640a98f0 2024-03-25T19:40:15.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.286+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535447266 0 0) 0x7f1564142520 con 0x7f156413f710 2024-03-25T19:40:15.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.286+0000 7f156911c640 10 monclient(hunting): _negotiate 2024-03-25T19:40:15.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.286+0000 7f156911c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:15.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.286+0000 7f156911c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:15.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.286+0000 7f156911c640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:40:15.287 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.286+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15640b28a0 con 0x7f156413f710 2024-03-25T19:40:15.289 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.289+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595360862 0 0) 0x7f15640b2a30 con 0x7f15641430c0 2024-03-25T19:40:15.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.289+0000 7f156911c640 10 monclient(hunting): _negotiate 2024-03-25T19:40:15.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.289+0000 7f156911c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:15.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.289+0000 7f156911c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:15.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.289+0000 7f156911c640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:40:15.290 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.289+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1564142520 con 0x7f15641430c0 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3504730045 0 0) 0x7f1550003520 con 0x7f15640a98f0 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 >> v1:172.21.15.143:6790/0 conn(0x7f15641430c0 legacy=0x7f156414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 >> v1:172.21.15.143:6789/0 conn(0x7f156413f710 legacy=0x7f15641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 10 monclient: _finish_hunting 0 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 1 monclient: found mon.noname-c 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15640b44c0 con 0x7f15640a98f0 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 10 monclient: _finish_auth 0 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 10 monclient: _check_auth_tickets 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:15.292 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1560001690 con 0x7f15640a98f0 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.291+0000 7f156911c640 10 monclient: dump: 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.292+0000 7f156c382640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:40:15.293 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.292+0000 7f156c382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.301+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15600017f0 con 0x7f15640a98f0 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.301+0000 7f156911c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.301+0000 7f156911c640 1 -- 172.21.15.143:0/1792090861 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f1560001c00 con 0x7f15640a98f0 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.301+0000 7f156911c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.301+0000 7f156911c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.301+0000 7f156911c640 10 monclient: dump: 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:15.301 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:15.302 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:15.302 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:15.302 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:15.302 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:15.302 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:15.302 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:15.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.301+0000 7f156c382640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:15.302 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.301+0000 7f156c382640 4 set_mon_vals no callback set 2024-03-25T19:40:15.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.302+0000 7f156c382640 10 monclient: shutdown 2024-03-25T19:40:15.303 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.302+0000 7f156c382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.302+0000 7f156c382640 1 -- 172.21.15.143:0/1792090861 >> v1:172.21.15.203:6789/0 conn(0x7f15640a98f0 legacy=0x7f15640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.302+0000 7f156c382640 1 -- 172.21.15.143:0/1792090861 shutdown_connections 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.302+0000 7f156c382640 1 -- 172.21.15.143:0/1792090861 >> 172.21.15.143:0/1792090861 conn(0x7f15640b7ea0 msgr2=0x7f15640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.302+0000 7f156c382640 1 -- 172.21.15.143:0/1792090861 shutdown_connections 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.302+0000 7f156c382640 1 -- 172.21.15.143:0/1792090861 wait complete. 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.303+0000 7f156c382640 10 monclient: build_initial_monmap 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.303+0000 7f156c382640 10 monclient: monmap: 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:15.304 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.303+0000 7f156c382640 1 librados: starting msgr at 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.303+0000 7f156c382640 1 librados: starting objecter 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 1 Processor -- start 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 1 -- start start 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 1 librados: setting wanted keys 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 1 librados: calling monclient init 2024-03-25T19:40:15.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 10 monclient: init 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 10 monclient(hunting): picked mon.noname-c con 0x7f15640a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 10 monclient(hunting): picked mon.noname-a con 0x7f156413f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 10 monclient(hunting): picked mon.noname-b con 0x7f15641430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1564136e30 con 0x7f156413f710 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.304+0000 7f156c382640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1564135210 con 0x7f15641430c0 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.305+0000 7f156c382640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1564135cb0 con 0x7f15640a98f0 2024-03-25T19:40:15.307 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.305+0000 7f156c382640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:15.308 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.305+0000 7f156c382640 10 monclient(hunting): authenticate will time out at 6631.648653s 2024-03-25T19:40:15.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.311+0000 7f156a91f640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f15641430c0 0x7f15641e2fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:34276/0 (socket says 172.21.15.143:34276) 2024-03-25T19:40:15.311 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.311+0000 7f156a91f640 1 -- 172.21.15.143:0/1024768654 learned_addr learned my addr 172.21.15.143:0/1024768654 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:15.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.312+0000 7f155affd640 10 client.?.objecter ms_handle_connect 0x7f15641430c0 2024-03-25T19:40:15.312 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.313+0000 7f155affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:15.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.329+0000 7f155affd640 10 client.?.objecter ms_handle_connect 0x7f156413f710 2024-03-25T19:40:15.328 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.329+0000 7f155affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:15.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.331+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452567083 0 0) 0x7f1564135210 con 0x7f15641430c0 2024-03-25T19:40:15.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.331+0000 7f155affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:15.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.331+0000 7f155affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:15.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.331+0000 7f155affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:15.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.331+0000 7f155affd640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:40:15.331 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.331+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1548004eb0 con 0x7f15641430c0 2024-03-25T19:40:15.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.337+0000 7f155affd640 10 client.?.objecter ms_handle_connect 0x7f15640a98f0 2024-03-25T19:40:15.337 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.337+0000 7f155affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:15.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.338+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3932198116 0 0) 0x7f1548004eb0 con 0x7f15641430c0 2024-03-25T19:40:15.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.338+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1564135210 con 0x7f15641430c0 2024-03-25T19:40:15.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.338+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f155c001c70 con 0x7f15641430c0 2024-03-25T19:40:15.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.338+0000 7f155affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:15.338 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.338+0000 7f155affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.338+0000 7f155affd640 10 monclient(hunting): dump: 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:15.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.338+0000 7f155affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:15.343 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.344+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81443046 0 0) 0x7f1564136e30 con 0x7f156413f710 2024-03-25T19:40:15.343 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.344+0000 7f155affd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:15.343 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.344+0000 7f155affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:15.343 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.344+0000 7f155affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:15.343 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.344+0000 7f155affd640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:40:15.343 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.344+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1548004eb0 con 0x7f156413f710 2024-03-25T19:40:15.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3368767525 0 0) 0x7f1564135210 con 0x7f15641430c0 2024-03-25T19:40:15.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 >> v1:172.21.15.143:6790/0 conn(0x7f15640a98f0 legacy=0x7f1564130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:15.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 >> v1:172.21.15.143:6789/0 conn(0x7f156413f710 legacy=0x7f15641dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:15.346 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 1 monclient: found mon.b 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1564135e40 con 0x7f15641430c0 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 10 monclient: _finish_auth 0 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f155affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 5 monclient: authenticate success, global_id 4696 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 10 monclient: _renew_subs 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1564131680 con 0x7f15641430c0 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 10 monclient: _renew_subs 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:15.347 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f156413b800 con 0x7f15641430c0 2024-03-25T19:40:15.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.346+0000 7f156c382640 1 librados: init done 2024-03-25T19:40:15.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.347+0000 7f156c382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:15.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.347+0000 7f156c382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:15.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.347+0000 7f156c382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:15.348 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.347+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156413d010 con 0x7f15641430c0 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.637+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f155c003990 con 0x7f15641430c0 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.637+0000 7f155affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.637+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f155c003d60 con 0x7f15641430c0 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.637+0000 7f155affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.637+0000 7f155affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.637+0000 7f155affd640 10 monclient: dump: 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:15.637 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:15.638 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:15.638 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:15.638 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:15.638 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:15.638 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:15.638 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.637+0000 7f156891b640 4 set_mon_vals no callback set 2024-03-25T19:40:15.638 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.638+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f155c003f80 con 0x7f15641430c0 2024-03-25T19:40:15.639 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.639+0000 7f156a11e640 10 monclient: get_auth_request con 0x7f154806dd90 auth_method 0 2024-03-25T19:40:15.639 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.639+0000 7f155affd640 10 client.4696.objecter ms_handle_connect 0x7f154806dd90 2024-03-25T19:40:15.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.643+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7f155c054a30 con 0x7f15641430c0 2024-03-25T19:40:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.643+0000 7f155affd640 10 client.4696.objecter ms_dispatch 0x7f1564000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.643+0000 7f155affd640 3 client.4696.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.643+0000 7f155affd640 3 client.4696.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.643+0000 7f155affd640 20 client.4696.objecter dump_active .. 0 homeless 2024-03-25T19:40:15.643 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.643+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f155c050540 con 0x7f15641430c0 2024-03-25T19:40:15.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.643+0000 7f155affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:15.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.643+0000 7f155affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:15.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.876+0000 7f156c382640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:15.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.876+0000 7f156c382640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:15.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.876+0000 7f156c382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:15.876 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.876+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15641372c0 con 0x7f15641430c0 2024-03-25T19:40:15.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.988+0000 7f155affd640 1 -- 172.21.15.143:0/1024768654 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7f15641372c0 con 0x7f15641430c0 2024-03-25T19:40:15.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.988+0000 7f155affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:15.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.988+0000 7f155affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:15.988 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:40:15.988 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.990+0000 7f156c382640 10 librados: watch_flush enter 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.990+0000 7f156c382640 10 librados: watch_flush exit 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 20 client.4696.objecter shutdown clearing up homeless session... 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 10 client.4696.objecter successfully canceled tick 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 >> v1:172.21.15.143:6800/1292282666 conn(0x7f154806dd90 legacy=0x7f1548070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 10 monclient: shutdown 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 >> v1:172.21.15.203:6789/0 conn(0x7f15641430c0 legacy=0x7f15641e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 shutdown_connections 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 >> 172.21.15.143:0/1024768654 conn(0x7f15640b7ea0 msgr2=0x7f1564074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:15.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 shutdown_connections 2024-03-25T19:40:15.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 1 -- 172.21.15.143:0/1024768654 wait complete. 2024-03-25T19:40:15.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:15.991+0000 7f156c382640 1 librados: shutdown 2024-03-25T19:40:16.002 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:19.751 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:46.315 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.757 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:46.322 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.794 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:10:46.358 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.821 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:46.357 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.821 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:46.386 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.826 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:46.362 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.831 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:46.396 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.853 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:10:46.389 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:19.870 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:10:46.435 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:22.809 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:22.909 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.910+0000 7fb1a2e63640 10 monclient: get_monmap_and_config 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.910+0000 7fb1a2e63640 10 monclient: build_initial_monmap 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.910+0000 7fb1a2e63640 10 monclient: monmap: 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:40:22.911044+0000 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:40:22.911044+0000 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:40:22.910 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:22.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 1 Processor -- start 2024-03-25T19:40:22.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 1 -- start start 2024-03-25T19:40:22.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 10 monclient: init 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 10 monclient(hunting): picked mon.noname-c con 0x7fb19c0a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 10 monclient(hunting): picked mon.noname-b con 0x7fb19c13f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 10 monclient(hunting): picked mon.noname-a con 0x7fb19c1430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c142520 con 0x7fb19c1430c0 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c0b28a0 con 0x7fb19c0a98f0 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c0b2a30 con 0x7fb19c13f710 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.911+0000 7fb1a2e63640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:22.912 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.912+0000 7fb1a2e63640 10 monclient(hunting): authenticate will time out at 6639.255652s 2024-03-25T19:40:22.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.926+0000 7fb1a0bff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb19c0a98f0 0x7fb19c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:32940/0 (socket says 172.21.15.143:32940) 2024-03-25T19:40:22.925 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.926+0000 7fb1a0bff640 1 -- 172.21.15.143:0/545511424 learned_addr learned my addr 172.21.15.143:0/545511424 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:22.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.942+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955727861 0 0) 0x7fb19c0b28a0 con 0x7fb19c0a98f0 2024-03-25T19:40:22.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.943+0000 7fb19b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.943+0000 7fb19b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.942 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.943+0000 7fb19b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.943+0000 7fb19b7fe640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:40:22.943 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.943+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb184003520 con 0x7fb19c0a98f0 2024-03-25T19:40:22.947 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.948+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108618144 0 0) 0x7fb19c0b2a30 con 0x7fb19c13f710 2024-03-25T19:40:22.947 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.948+0000 7fb19b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.947 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.948+0000 7fb19b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.948+0000 7fb19b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.948+0000 7fb19b7fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:40:22.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.948+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19c0b28a0 con 0x7fb19c13f710 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107263149 0 0) 0x7fb19c142520 con 0x7fb19c1430c0 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19c0b2a30 con 0x7fb19c1430c0 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3301874570 0 0) 0x7fb184003520 con 0x7fb19c0a98f0 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 >> v1:172.21.15.143:6790/0 conn(0x7fb19c13f710 legacy=0x7fb19c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 >> v1:172.21.15.143:6789/0 conn(0x7fb19c1430c0 legacy=0x7fb19c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:40:22.956 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c0b44c0 con 0x7fb19c0a98f0 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb18c001a10 con 0x7fb19c0a98f0 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb19b7fe640 10 monclient: dump: 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:22.957 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:22.958 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:22.958 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:22.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb1a2e63640 5 monclient: authenticate success, global_id 4699 2024-03-25T19:40:22.958 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.955+0000 7fb1a2e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:22.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.974+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb18c001b70 con 0x7fb19c0a98f0 2024-03-25T19:40:22.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.974+0000 7fb19b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:22.973 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.974+0000 7fb19b7fe640 1 -- 172.21.15.143:0/545511424 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb18c001f80 con 0x7fb19c0a98f0 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.974+0000 7fb19b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.974+0000 7fb19b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.974+0000 7fb19b7fe640 10 monclient: dump: 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.974+0000 7fb1a2e63640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:22.974 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.974+0000 7fb1a2e63640 4 set_mon_vals no callback set 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.975+0000 7fb1a2e63640 10 monclient: shutdown 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.975+0000 7fb1a2e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.975+0000 7fb1a2e63640 1 -- 172.21.15.143:0/545511424 >> v1:172.21.15.203:6789/0 conn(0x7fb19c0a98f0 legacy=0x7fb19c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.975+0000 7fb1a2e63640 1 -- 172.21.15.143:0/545511424 shutdown_connections 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.975+0000 7fb1a2e63640 1 -- 172.21.15.143:0/545511424 >> 172.21.15.143:0/545511424 conn(0x7fb19c0b7ea0 msgr2=0x7fb19c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.975+0000 7fb1a2e63640 1 -- 172.21.15.143:0/545511424 shutdown_connections 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.975+0000 7fb1a2e63640 1 -- 172.21.15.143:0/545511424 wait complete. 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 10 monclient: build_initial_monmap 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 10 monclient: monmap: 2024-03-25T19:40:22.976 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 1 librados: starting msgr at 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 1 librados: starting objecter 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 1 Processor -- start 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 1 -- start start 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 1 librados: setting wanted keys 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 1 librados: calling monclient init 2024-03-25T19:40:22.977 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.976+0000 7fb1a2e63640 10 monclient: init 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 10 monclient(hunting): picked mon.noname-a con 0x7fb19c0a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 10 monclient(hunting): picked mon.noname-c con 0x7fb19c13f710 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 10 monclient(hunting): picked mon.noname-b con 0x7fb19c1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c138250 con 0x7fb19c0a98f0 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c137d50 con 0x7fb19c1430c0 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19c137ee0 con 0x7fb19c13f710 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:22.979 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.977+0000 7fb1a2e63640 10 monclient(hunting): authenticate will time out at 6639.321469s 2024-03-25T19:40:22.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.986+0000 7fb1a0bff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fb19c0a98f0 0x7fb19c137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:47456/0 (socket says 172.21.15.143:47456) 2024-03-25T19:40:22.985 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.986+0000 7fb1a0bff640 1 -- 172.21.15.143:0/1629853018 learned_addr learned my addr 172.21.15.143:0/1629853018 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:22.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.988+0000 7fb1997fa640 10 client.?.objecter ms_handle_connect 0x7fb19c1430c0 2024-03-25T19:40:22.988 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.988+0000 7fb1997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.992+0000 7fb1997fa640 10 client.?.objecter ms_handle_connect 0x7fb19c0a98f0 2024-03-25T19:40:22.992 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.992+0000 7fb1997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.996+0000 7fb1997fa640 10 client.?.objecter ms_handle_connect 0x7fb19c13f710 2024-03-25T19:40:22.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.996+0000 7fb1997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:22.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.996+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290448420 0 0) 0x7fb19c138250 con 0x7fb19c0a98f0 2024-03-25T19:40:22.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.996+0000 7fb1997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.996+0000 7fb1997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.996+0000 7fb1997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.996+0000 7fb1997fa640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:40:22.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.996+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb188004eb0 con 0x7fb19c0a98f0 2024-03-25T19:40:22.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.999+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272652618 0 0) 0x7fb19c137d50 con 0x7fb19c1430c0 2024-03-25T19:40:22.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.999+0000 7fb1997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:22.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.999+0000 7fb1997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:22.998 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.999+0000 7fb1997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:22.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.999+0000 7fb1997fa640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:40:22.999 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:22.999+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb19c138250 con 0x7fb19c1430c0 2024-03-25T19:40:23.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.011+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2862140636 0 0) 0x7fb188004eb0 con 0x7fb19c0a98f0 2024-03-25T19:40:23.010 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.011+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb19c137d50 con 0x7fb19c0a98f0 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.011+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb18c001f40 con 0x7fb19c0a98f0 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.011+0000 7fb1997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.011+0000 7fb1997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.011+0000 7fb1997fa640 10 monclient(hunting): dump: 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:23.011 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.011+0000 7fb1997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:23.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.022+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893136328 0 0) 0x7fb19c137ee0 con 0x7fb19c13f710 2024-03-25T19:40:23.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.022+0000 7fb1997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:23.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.022+0000 7fb1997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:23.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.022+0000 7fb1997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:23.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.022+0000 7fb1997fa640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:40:23.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.022+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb188004eb0 con 0x7fb19c13f710 2024-03-25T19:40:23.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 305506772 0 0) 0x7fb19c137d50 con 0x7fb19c0a98f0 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 >> v1:172.21.15.143:6790/0 conn(0x7fb19c13f710 legacy=0x7fb19c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 >> v1:172.21.15.203:6789/0 conn(0x7fb19c1430c0 legacy=0x7fb19c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 1 monclient: found mon.a 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c0b7730 con 0x7fb19c0a98f0 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 10 monclient: _renew_subs 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:23.024 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb19c0b7920 con 0x7fb19c0a98f0 2024-03-25T19:40:23.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:23.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 10 monclient: _renew_subs 2024-03-25T19:40:23.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:23.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19c1e33d0 con 0x7fb19c0a98f0 2024-03-25T19:40:23.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.024+0000 7fb1a2e63640 1 librados: init done 2024-03-25T19:40:23.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.025+0000 7fb1a2e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.025+0000 7fb1a2e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.025+0000 7fb1a2e63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:23.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.025+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19c0627b0 con 0x7fb19c0a98f0 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.043+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb18c001570 con 0x7fb19c0a98f0 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.043+0000 7fb1997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.043+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb18c005030 con 0x7fb19c0a98f0 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.043+0000 7fb1997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.043+0000 7fb1997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.043+0000 7fb1997fa640 10 monclient: dump: 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:23.043 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:23.044 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:23.044 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:23.044 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:23.044 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:23.044 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:23.044 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.043+0000 7fb19affd640 4 set_mon_vals no callback set 2024-03-25T19:40:23.044 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.044+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fb18c01b080 con 0x7fb19c0a98f0 2024-03-25T19:40:23.044 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.045+0000 7fb19bfff640 10 monclient: get_auth_request con 0x7fb18806dd60 auth_method 0 2024-03-25T19:40:23.045 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.045+0000 7fb1997fa640 10 client.4695.objecter ms_handle_connect 0x7fb18806dd60 2024-03-25T19:40:23.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.048+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7fb18c054cf0 con 0x7fb19c0a98f0 2024-03-25T19:40:23.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.048+0000 7fb1997fa640 10 client.4695.objecter ms_dispatch 0x7fb19c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:23.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.048+0000 7fb1997fa640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:23.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.048+0000 7fb1997fa640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:23.048 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.048+0000 7fb1997fa640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T19:40:23.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.345+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb18c050630 con 0x7fb19c0a98f0 2024-03-25T19:40:23.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.345+0000 7fb1997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:23.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.345+0000 7fb1997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:23.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.592+0000 7fb1a2e63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:23.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.592+0000 7fb1a2e63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:23.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.592+0000 7fb1a2e63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:23.592 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.592+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb19c13df50 con 0x7fb19c0a98f0 2024-03-25T19:40:23.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.619+0000 7fb1997fa640 1 -- 172.21.15.143:0/1629853018 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7fb18c052730 con 0x7fb19c0a98f0 2024-03-25T19:40:23.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.619+0000 7fb1997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:23.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.619+0000 7fb1997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:23.619 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:40:23.619 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:23.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 10 librados: watch_flush enter 2024-03-25T19:40:23.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 10 librados: watch_flush exit 2024-03-25T19:40:23.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T19:40:23.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 10 client.4695.objecter successfully canceled tick 2024-03-25T19:40:23.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 >> v1:172.21.15.143:6800/1292282666 conn(0x7fb18806dd60 legacy=0x7fb188070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 10 monclient: shutdown 2024-03-25T19:40:23.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:23.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 >> v1:172.21.15.143:6789/0 conn(0x7fb19c0a98f0 legacy=0x7fb19c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:23.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 shutdown_connections 2024-03-25T19:40:23.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 >> 172.21.15.143:0/1629853018 conn(0x7fb19c0b7ea0 msgr2=0x7fb19c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:23.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 shutdown_connections 2024-03-25T19:40:23.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 1 -- 172.21.15.143:0/1629853018 wait complete. 2024-03-25T19:40:23.622 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:23.621+0000 7fb1a2e63640 1 librados: shutdown 2024-03-25T19:40:23.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:30.441 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:30.559 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.559+0000 7fab449cf640 10 monclient: get_monmap_and_config 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.559+0000 7fab449cf640 10 monclient: build_initial_monmap 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.559+0000 7fab449cf640 10 monclient: monmap: 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:40:30.560504+0000 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:40:30.560504+0000 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:30.560 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:30.561 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:40:30.561 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:40:30.561 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:30.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.560+0000 7fab449cf640 1 Processor -- start 2024-03-25T19:40:30.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.560+0000 7fab449cf640 1 -- start start 2024-03-25T19:40:30.561 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.560+0000 7fab449cf640 10 monclient: init 2024-03-25T19:40:30.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:30.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:30.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 10 monclient(hunting): picked mon.noname-c con 0x7fab400b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:30.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 10 monclient(hunting): picked mon.noname-b con 0x7fab400b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:30.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 10 monclient(hunting): picked mon.noname-a con 0x7fab4014be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:30.562 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab400af940 con 0x7fab4014be90 2024-03-25T19:40:30.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab400b0a70 con 0x7fab400b4d60 2024-03-25T19:40:30.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab400b0d50 con 0x7fab400b1730 2024-03-25T19:40:30.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:30.563 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.561+0000 7fab449cf640 10 monclient(hunting): authenticate will time out at 6646.904814s 2024-03-25T19:40:30.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.570+0000 7fab3ed9d640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fab4014be90 0x7fab4014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51716/0 (socket says 172.21.15.143:51716) 2024-03-25T19:40:30.569 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.570+0000 7fab3ed9d640 1 -- 172.21.15.143:0/1334479787 learned_addr learned my addr 172.21.15.143:0/1334479787 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:30.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.586+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611348002 0 0) 0x7fab400af940 con 0x7fab4014be90 2024-03-25T19:40:30.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.586+0000 7fab3d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:30.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.586+0000 7fab3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:30.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.586+0000 7fab3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:30.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.586+0000 7fab3d59a640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:40:30.586 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.586+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab28003520 con 0x7fab4014be90 2024-03-25T19:40:30.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.587+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979194974 0 0) 0x7fab400b0a70 con 0x7fab400b4d60 2024-03-25T19:40:30.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.587+0000 7fab3d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:30.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.587+0000 7fab3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:30.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.587+0000 7fab3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:30.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.587+0000 7fab3d59a640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:40:30.587 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.587+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab400af940 con 0x7fab400b4d60 2024-03-25T19:40:30.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.595+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659154161 0 0) 0x7fab400b0d50 con 0x7fab400b1730 2024-03-25T19:40:30.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.595+0000 7fab3d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:40:30.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.595+0000 7fab3d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:30.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.595+0000 7fab3d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:30.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.595+0000 7fab3d59a640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:40:30.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.595+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab400b0a70 con 0x7fab400b1730 2024-03-25T19:40:30.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441768191 0 0) 0x7fab400af940 con 0x7fab400b4d60 2024-03-25T19:40:30.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 >> v1:172.21.15.143:6790/0 conn(0x7fab400b1730 legacy=0x7fab400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:30.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 >> v1:172.21.15.143:6789/0 conn(0x7fab4014be90 legacy=0x7fab4014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:30.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:40:30.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 1 monclient: found mon.noname-c 2024-03-25T19:40:30.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:30.597 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab400b1030 con 0x7fab400b4d60 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 10 monclient: _finish_auth 0 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fab30001670 con 0x7fab400b4d60 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab449cf640 5 monclient: authenticate success, global_id 4705 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab449cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab3d59a640 10 monclient: dump: 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:30.598 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:30.599 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.597+0000 7fab449cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:30.607 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.608+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab300017d0 con 0x7fab400b4d60 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.608+0000 7fab3d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.608+0000 7fab3d59a640 1 -- 172.21.15.143:0/1334479787 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fab30001be0 con 0x7fab400b4d60 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.608+0000 7fab3d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.608+0000 7fab3d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.608+0000 7fab3d59a640 10 monclient: dump: 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:30.608 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:30.609 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:30.609 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:30.609 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:30.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.608+0000 7fab449cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:30.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.608+0000 7fab449cf640 4 set_mon_vals no callback set 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.609+0000 7fab449cf640 10 monclient: shutdown 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.609+0000 7fab449cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.609+0000 7fab449cf640 1 -- 172.21.15.143:0/1334479787 >> v1:172.21.15.203:6789/0 conn(0x7fab400b4d60 legacy=0x7fab4013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.609+0000 7fab449cf640 1 -- 172.21.15.143:0/1334479787 shutdown_connections 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.609+0000 7fab449cf640 1 -- 172.21.15.143:0/1334479787 >> 172.21.15.143:0/1334479787 conn(0x7fab400617a0 msgr2=0x7fab40061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.609+0000 7fab449cf640 1 -- 172.21.15.143:0/1334479787 shutdown_connections 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.609+0000 7fab449cf640 1 -- 172.21.15.143:0/1334479787 wait complete. 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.610+0000 7fab449cf640 10 monclient: build_initial_monmap 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.610+0000 7fab449cf640 10 monclient: monmap: 2024-03-25T19:40:30.611 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.610+0000 7fab449cf640 1 librados: starting msgr at 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.610+0000 7fab449cf640 1 librados: starting objecter 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.610+0000 7fab449cf640 1 Processor -- start 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.610+0000 7fab449cf640 1 -- start start 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.610+0000 7fab449cf640 1 librados: setting wanted keys 2024-03-25T19:40:30.612 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.611+0000 7fab449cf640 1 librados: calling monclient init 2024-03-25T19:40:30.613 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.611+0000 7fab449cf640 10 monclient: init 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.611+0000 7fab449cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 10 monclient(hunting): picked mon.noname-a con 0x7fab400b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 10 monclient(hunting): picked mon.noname-c con 0x7fab400b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 10 monclient(hunting): picked mon.noname-b con 0x7fab4014be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab40142f20 con 0x7fab400b1730 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab40141c70 con 0x7fab4014be90 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab40141e00 con 0x7fab400b4d60 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:30.614 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.612+0000 7fab449cf640 10 monclient(hunting): authenticate will time out at 6646.955992s 2024-03-25T19:40:30.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.626+0000 7fab3e59c640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7fab400b1730 0x7fab401426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51736/0 (socket says 172.21.15.143:51736) 2024-03-25T19:40:30.626 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.626+0000 7fab3e59c640 1 -- 172.21.15.143:0/4096719494 learned_addr learned my addr 172.21.15.143:0/4096719494 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:30.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.628+0000 7fab2effd640 10 client.?.objecter ms_handle_connect 0x7fab4014be90 2024-03-25T19:40:30.627 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.628+0000 7fab2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:30.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.629+0000 7fab2effd640 10 client.?.objecter ms_handle_connect 0x7fab400b1730 2024-03-25T19:40:30.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.629+0000 7fab2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:30.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.634+0000 7fab2effd640 10 client.?.objecter ms_handle_connect 0x7fab400b4d60 2024-03-25T19:40:30.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.634+0000 7fab2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:30.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.641+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349975383 0 0) 0x7fab40141c70 con 0x7fab4014be90 2024-03-25T19:40:30.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.641+0000 7fab2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:30.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.641+0000 7fab2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:30.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.641+0000 7fab2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:30.641 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.641+0000 7fab2effd640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:40:30.642 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.641+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab20004eb0 con 0x7fab4014be90 2024-03-25T19:40:30.655 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.656+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207698979 0 0) 0x7fab40142f20 con 0x7fab400b1730 2024-03-25T19:40:30.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.656+0000 7fab2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:30.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.656+0000 7fab2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:30.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.656+0000 7fab2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:30.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.656+0000 7fab2effd640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:40:30.656 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.656+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab40141c70 con 0x7fab400b1730 2024-03-25T19:40:30.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.659+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374434321 0 0) 0x7fab40141e00 con 0x7fab400b4d60 2024-03-25T19:40:30.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.659+0000 7fab2effd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:30.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.659+0000 7fab2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:30.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.659+0000 7fab2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:30.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.659+0000 7fab2effd640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:40:30.659 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.659+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab40142f20 con 0x7fab400b4d60 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.674+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2605698475 0 0) 0x7fab20004eb0 con 0x7fab4014be90 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.674+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab40141e00 con 0x7fab4014be90 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.674+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fab38001af0 con 0x7fab4014be90 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.674+0000 7fab2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.674+0000 7fab2effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.674+0000 7fab2effd640 10 monclient(hunting): dump: 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:30.674 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:30.675 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:30.675 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:30.675 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:30.675 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:30.675 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.674+0000 7fab2effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:30.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.678+0000 7fab3ed9d640 1 -- 172.21.15.143:0/4096719494 >> v1:172.21.15.203:6789/0 conn(0x7fab4014be90 legacy=0x7fab401d8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:40:30.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.678+0000 7fab3ed9d640 1 -- 172.21.15.143:0/4096719494 >> v1:172.21.15.203:6789/0 conn(0x7fab4014be90 legacy=0x7fab401d8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:40:30.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.678+0000 7fab3ed9d640 1 --1- 172.21.15.143:0/4096719494 >> v1:172.21.15.203:6789/0 conn(0x7fab4014be90 0x7fab401d8040 :-1 s=OPENED pgs=80869 cs=1 l=1).handle_message read tag failed 2024-03-25T19:40:30.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.678+0000 7fab3ed9d640 1 --1- 172.21.15.143:0/4096719494 >> v1:172.21.15.203:6789/0 conn(0x7fab4014be90 0x7fab401d8040 :-1 s=OPENED pgs=80869 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:40:30.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.678+0000 7fab2effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.203:6789/0 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.705+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477835674 0 0) 0x7fab40141c70 con 0x7fab400b1730 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.705+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab20004eb0 con 0x7fab400b1730 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.705+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fab30001ba0 con 0x7fab400b1730 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.705+0000 7fab2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.705+0000 7fab2effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.705+0000 7fab2effd640 10 monclient(hunting): dump: 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:30.705 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:30.706 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:30.706 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:30.706 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:30.706 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:30.706 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:30.706 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:30.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1183842763 0 0) 0x7fab20004eb0 con 0x7fab400b1730 2024-03-25T19:40:30.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 >> v1:172.21.15.143:6790/0 conn(0x7fab400b4d60 legacy=0x7fab401d7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:30.784 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 >> v1:172.21.15.203:6789/0 conn(0x7fab4014be90 legacy=0x7fab401d8040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 1 monclient: found mon.a 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab40141f90 con 0x7fab400b1730 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 10 monclient: _finish_auth 0 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 5 monclient: authenticate success, global_id 4701 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 10 monclient: _renew_subs 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:30.785 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4013e270 con 0x7fab400b1730 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 10 client.4701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 10 monclient: _renew_subs 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4013e610 con 0x7fab400b1730 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.784+0000 7fab449cf640 1 librados: init done 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.785+0000 7fab449cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.785+0000 7fab449cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.785+0000 7fab449cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:30.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.785+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab400627b0 con 0x7fab400b1730 2024-03-25T19:40:30.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.797+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab300038a0 con 0x7fab400b1730 2024-03-25T19:40:30.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.797+0000 7fab2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:30.797 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:30.797+0000 7fab3cd99640 4 set_mon_vals no callback set 2024-03-25T19:40:31.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.083+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fab30003cb0 con 0x7fab400b1730 2024-03-25T19:40:31.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.083+0000 7fab2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:31.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.083+0000 7fab2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:31.083 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.083+0000 7fab2effd640 10 monclient: dump: 2024-03-25T19:40:31.083 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:31.083 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:31.083 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:31.084 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:31.084 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:31.084 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:31.084 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:31.084 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:31.084 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:31.084 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:31.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.084+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fab30003e90 con 0x7fab400b1730 2024-03-25T19:40:31.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.085+0000 7fab3dd9b640 10 monclient: get_auth_request con 0x7fab2006df70 auth_method 0 2024-03-25T19:40:31.085 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.085+0000 7fab2effd640 10 client.4701.objecter ms_handle_connect 0x7fab2006df70 2024-03-25T19:40:31.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.090+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7fab30054880 con 0x7fab400b1730 2024-03-25T19:40:31.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.090+0000 7fab2effd640 10 client.4701.objecter ms_dispatch 0x7fab40000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:31.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.090+0000 7fab2effd640 3 client.4701.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:31.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.090+0000 7fab2effd640 3 client.4701.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:31.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.090+0000 7fab2effd640 20 client.4701.objecter dump_active .. 0 homeless 2024-03-25T19:40:31.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.090+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab30050390 con 0x7fab400b1730 2024-03-25T19:40:31.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.090+0000 7fab2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:31.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.090+0000 7fab2effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:31.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.339+0000 7fab449cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:31.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.339+0000 7fab449cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:31.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.339+0000 7fab449cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:31.339 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.339+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab4013e400 con 0x7fab400b1730 2024-03-25T19:40:31.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.405+0000 7fab2effd640 1 -- 172.21.15.143:0/4096719494 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7fab300522c0 con 0x7fab400b1730 2024-03-25T19:40:31.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.405+0000 7fab2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:31.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.405+0000 7fab2effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:31.405 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:40:31.405 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 10 librados: watch_flush enter 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 10 librados: watch_flush exit 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 20 client.4701.objecter shutdown clearing up homeless session... 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 10 client.4701.objecter successfully canceled tick 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 >> v1:172.21.15.143:6800/1292282666 conn(0x7fab2006df70 legacy=0x7fab20070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 10 monclient: shutdown 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 >> v1:172.21.15.143:6789/0 conn(0x7fab400b1730 legacy=0x7fab401426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 shutdown_connections 2024-03-25T19:40:31.408 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 >> 172.21.15.143:0/4096719494 conn(0x7fab400617a0 msgr2=0x7fab4013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:31.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 shutdown_connections 2024-03-25T19:40:31.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 1 -- 172.21.15.143:0/4096719494 wait complete. 2024-03-25T19:40:31.409 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:31.408+0000 7fab449cf640 1 librados: shutdown 2024-03-25T19:40:31.420 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:38.228 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:38.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.342+0000 7fb0b10b3640 10 monclient: get_monmap_and_config 2024-03-25T19:40:38.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.342+0000 7fb0b10b3640 10 monclient: build_initial_monmap 2024-03-25T19:40:38.342 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.342+0000 7fb0b10b3640 10 monclient: monmap: 2024-03-25T19:40:38.342 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:40:38.343527+0000 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:40:38.343527+0000 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:40:38.343 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:38.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.343+0000 7fb0b10b3640 1 Processor -- start 2024-03-25T19:40:38.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.343+0000 7fb0b10b3640 1 -- start start 2024-03-25T19:40:38.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.343+0000 7fb0b10b3640 10 monclient: init 2024-03-25T19:40:38.344 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ac0a98f0 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ac13f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ac1430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac142520 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0b28a0 con 0x7fb0ac13f710 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0b2a30 con 0x7fb0ac1430c0 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:38.345 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.344+0000 7fb0b10b3640 10 monclient(hunting): authenticate will time out at 6654.688028s 2024-03-25T19:40:38.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.357+0000 7fb0aa59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb0ac13f710 0x7fb0ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:54074/0 (socket says 172.21.15.143:54074) 2024-03-25T19:40:38.357 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.357+0000 7fb0aa59c640 1 -- 172.21.15.143:0/587492800 learned_addr learned my addr 172.21.15.143:0/587492800 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:38.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.369+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984280358 0 0) 0x7fb0ac0b28a0 con 0x7fb0ac13f710 2024-03-25T19:40:38.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.369+0000 7fb0a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:38.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.369+0000 7fb0a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:38.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.369+0000 7fb0a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:38.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.369+0000 7fb0a9d9b640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:40:38.369 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.369+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb094003520 con 0x7fb0ac13f710 2024-03-25T19:40:38.370 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.370+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914818927 0 0) 0x7fb0ac142520 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.370 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.370+0000 7fb0a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:38.370 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.370+0000 7fb0a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:38.370 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.370+0000 7fb0a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:38.370 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.370+0000 7fb0a9d9b640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:40:38.370 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.370+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ac0b28a0 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.385 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.386+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177787823 0 0) 0x7fb0ac0b2a30 con 0x7fb0ac1430c0 2024-03-25T19:40:38.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.386+0000 7fb0a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:40:38.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.386+0000 7fb0a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:38.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.386+0000 7fb0a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:38.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.386+0000 7fb0a9d9b640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:40:38.386 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.386+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ac142520 con 0x7fb0ac1430c0 2024-03-25T19:40:38.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236124715 0 0) 0x7fb0ac0b28a0 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 >> v1:172.21.15.143:6790/0 conn(0x7fb0ac1430c0 legacy=0x7fb0ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:38.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 >> v1:172.21.15.203:6789/0 conn(0x7fb0ac13f710 legacy=0x7fb0ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:38.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:40:38.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 1 monclient: found mon.noname-a 2024-03-25T19:40:38.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:38.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:40:38.387 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac0b44c0 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 10 monclient: _finish_auth 0 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb09c001670 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0a9d9b640 10 monclient: dump: 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:38.388 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:38.389 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:38.389 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:38.389 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:38.389 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:38.389 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0b10b3640 5 monclient: authenticate success, global_id 4704 2024-03-25T19:40:38.389 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.387+0000 7fb0b10b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.401+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb09c0017d0 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.401+0000 7fb0a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.401+0000 7fb0a9d9b640 1 -- 172.21.15.143:0/587492800 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb09c001ba0 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.401+0000 7fb0a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.401+0000 7fb0a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.401+0000 7fb0a9d9b640 10 monclient: dump: 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.401 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:38.402 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:38.402 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:38.402 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:38.402 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:38.402 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:38.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.401+0000 7fb0b10b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:38.402 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.401+0000 7fb0b10b3640 4 set_mon_vals no callback set 2024-03-25T19:40:38.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.402+0000 7fb0b10b3640 10 monclient: shutdown 2024-03-25T19:40:38.403 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.402+0000 7fb0b10b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.402+0000 7fb0b10b3640 1 -- 172.21.15.143:0/587492800 >> v1:172.21.15.143:6789/0 conn(0x7fb0ac0a98f0 legacy=0x7fb0ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.402+0000 7fb0b10b3640 1 -- 172.21.15.143:0/587492800 shutdown_connections 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.402+0000 7fb0b10b3640 1 -- 172.21.15.143:0/587492800 >> 172.21.15.143:0/587492800 conn(0x7fb0ac0b7ea0 msgr2=0x7fb0ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.402+0000 7fb0b10b3640 1 -- 172.21.15.143:0/587492800 shutdown_connections 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.402+0000 7fb0b10b3640 1 -- 172.21.15.143:0/587492800 wait complete. 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.403+0000 7fb0b10b3640 10 monclient: build_initial_monmap 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.403+0000 7fb0b10b3640 10 monclient: monmap: 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:38.404 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.404+0000 7fb0b10b3640 1 librados: starting msgr at 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.404+0000 7fb0b10b3640 1 librados: starting objecter 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.404+0000 7fb0b10b3640 1 Processor -- start 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.404+0000 7fb0b10b3640 1 -- start start 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.404+0000 7fb0b10b3640 1 librados: setting wanted keys 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.404+0000 7fb0b10b3640 1 librados: calling monclient init 2024-03-25T19:40:38.405 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.404+0000 7fb0b10b3640 10 monclient: init 2024-03-25T19:40:38.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:38.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:38.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb0ac0a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:38.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb0ac13f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:38.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb0ac1430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:38.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac138f10 con 0x7fb0ac13f710 2024-03-25T19:40:38.406 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac139970 con 0x7fb0ac1430c0 2024-03-25T19:40:38.407 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac0abb50 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.407 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:38.407 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.405+0000 7fb0b10b3640 10 monclient(hunting): authenticate will time out at 6654.749028s 2024-03-25T19:40:38.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.423+0000 7fb0ab59e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb0ac1430c0 0x7fb0ac138320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:54098/0 (socket says 172.21.15.143:54098) 2024-03-25T19:40:38.422 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.423+0000 7fb0ab59e640 1 -- 172.21.15.143:0/4026980662 learned_addr learned my addr 172.21.15.143:0/4026980662 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:38.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.425+0000 7fb0937fe640 10 client.?.objecter ms_handle_connect 0x7fb0ac1430c0 2024-03-25T19:40:38.425 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.426+0000 7fb0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:38.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.429+0000 7fb0937fe640 10 client.?.objecter ms_handle_connect 0x7fb0ac0a98f0 2024-03-25T19:40:38.429 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.429+0000 7fb0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:38.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.431+0000 7fb0937fe640 10 client.?.objecter ms_handle_connect 0x7fb0ac13f710 2024-03-25T19:40:38.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.431+0000 7fb0937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:38.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.437+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336850676 0 0) 0x7fb0ac139970 con 0x7fb0ac1430c0 2024-03-25T19:40:38.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.437+0000 7fb0937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:38.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.437+0000 7fb0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:38.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.437+0000 7fb0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:38.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.437+0000 7fb0937fe640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:40:38.437 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.437+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb098004eb0 con 0x7fb0ac1430c0 2024-03-25T19:40:38.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.441+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911740779 0 0) 0x7fb0ac0abb50 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.441+0000 7fb0937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:38.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.441+0000 7fb0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:38.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.441+0000 7fb0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:38.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.441+0000 7fb0937fe640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:40:38.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.441+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ac139970 con 0x7fb0ac0a98f0 2024-03-25T19:40:38.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.448+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686092668 0 0) 0x7fb0ac138f10 con 0x7fb0ac13f710 2024-03-25T19:40:38.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.448+0000 7fb0937fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:38.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.448+0000 7fb0937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:38.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.448+0000 7fb0937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:38.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.448+0000 7fb0937fe640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:40:38.448 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.448+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ac0abb50 con 0x7fb0ac13f710 2024-03-25T19:40:38.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.452+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 929440603 0 0) 0x7fb098004eb0 con 0x7fb0ac1430c0 2024-03-25T19:40:38.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.452+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0ac138f10 con 0x7fb0ac1430c0 2024-03-25T19:40:38.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.452+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb0a4001d00 con 0x7fb0ac1430c0 2024-03-25T19:40:38.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.452+0000 7fb0937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.452+0000 7fb0937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.452+0000 7fb0937fe640 10 monclient(hunting): dump: 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:38.453 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.452+0000 7fb0937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:38.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.467+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 914891142 0 0) 0x7fb0ac0abb50 con 0x7fb0ac13f710 2024-03-25T19:40:38.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.467+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb098004eb0 con 0x7fb0ac13f710 2024-03-25T19:40:38.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.467+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb0a00017b0 con 0x7fb0ac13f710 2024-03-25T19:40:38.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.467+0000 7fb0937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:38.467 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.467+0000 7fb0937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.467+0000 7fb0937fe640 10 monclient(hunting): dump: 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:38.469 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:38.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2676987271 0 0) 0x7fb0ac138f10 con 0x7fb0ac1430c0 2024-03-25T19:40:38.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 >> v1:172.21.15.143:6790/0 conn(0x7fb0ac0a98f0 legacy=0x7fb0ac136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:38.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 >> v1:172.21.15.143:6789/0 conn(0x7fb0ac13f710 legacy=0x7fb0ac0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:38.470 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 1 monclient: found mon.b 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac0abe30 con 0x7fb0ac1430c0 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 5 monclient: authenticate success, global_id 4714 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 10 monclient: _renew_subs 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac137b40 con 0x7fb0ac1430c0 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 10 monclient: _renew_subs 2024-03-25T19:40:38.471 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:38.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac138c10 con 0x7fb0ac1430c0 2024-03-25T19:40:38.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.470+0000 7fb0b10b3640 1 librados: init done 2024-03-25T19:40:38.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.472+0000 7fb0b10b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:38.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.472+0000 7fb0b10b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:38.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.472+0000 7fb0b10b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:38.472 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.472+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac0627b0 con 0x7fb0ac1430c0 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.752+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0a4003990 con 0x7fb0ac1430c0 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.752+0000 7fb0937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.752+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7fb0a4003da0 con 0x7fb0ac1430c0 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.752+0000 7fb0937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.752+0000 7fb0937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.752+0000 7fb0937fe640 10 monclient: dump: 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.752 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:38.753 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:38.753 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:38.753 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:38.753 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:38.753 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:38.753 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:38.753 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.752+0000 7fb0a959a640 4 set_mon_vals no callback set 2024-03-25T19:40:38.754 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.753+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7fb0a4003fc0 con 0x7fb0ac1430c0 2024-03-25T19:40:38.754 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.753+0000 7fb0aad9d640 10 monclient: get_auth_request con 0x7fb09806df70 auth_method 0 2024-03-25T19:40:38.754 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.753+0000 7fb0937fe640 10 client.4714.objecter ms_handle_connect 0x7fb09806df70 2024-03-25T19:40:38.757 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.758+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7fb0a4054bb0 con 0x7fb0ac1430c0 2024-03-25T19:40:38.758 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.758+0000 7fb0937fe640 10 client.4714.objecter ms_dispatch 0x7fb0ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:38.758 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.758+0000 7fb0937fe640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:38.758 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.758+0000 7fb0937fe640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:38.758 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.758+0000 7fb0937fe640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T19:40:38.758 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.758+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0a40506c0 con 0x7fb0ac1430c0 2024-03-25T19:40:38.758 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.758+0000 7fb0937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:38.758 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.758+0000 7fb0937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:38.990 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.991+0000 7fb0b10b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:38.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.991+0000 7fb0b10b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:38.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.991+0000 7fb0b10b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:40:38.991 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:38.991+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0ac1390c0 con 0x7fb0ac1430c0 2024-03-25T19:40:39.022 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.023+0000 7fb0937fe640 1 -- 172.21.15.143:0/4026980662 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7fb0a40525f0 con 0x7fb0ac1430c0 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.023+0000 7fb0937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.023+0000 7fb0937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:40:39.023 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:39.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 10 librados: watch_flush enter 2024-03-25T19:40:39.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 10 librados: watch_flush exit 2024-03-25T19:40:39.025 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 10 client.4714.objecter successfully canceled tick 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 >> v1:172.21.15.143:6800/1292282666 conn(0x7fb09806df70 legacy=0x7fb098070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 10 monclient: shutdown 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 >> v1:172.21.15.203:6789/0 conn(0x7fb0ac1430c0 legacy=0x7fb0ac138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 shutdown_connections 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 >> 172.21.15.143:0/4026980662 conn(0x7fb0ac0b7ea0 msgr2=0x7fb0ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 shutdown_connections 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 1 -- 172.21.15.143:0/4026980662 wait complete. 2024-03-25T19:40:39.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:39.025+0000 7fb0b10b3640 1 librados: shutdown 2024-03-25T19:40:39.037 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:44.804 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:11.369 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:44.852 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:11.417 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:44.883 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:11.448 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:44.910 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:11.475 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:44.921 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:11.457 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:44.923 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:11.488 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:44.925 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:11.460 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:44.954 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:11.519 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:44.957 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:11.493 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:45.843 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:45.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.946+0000 7f880ec7b640 10 monclient: get_monmap_and_config 2024-03-25T19:40:45.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.946+0000 7f880ec7b640 10 monclient: build_initial_monmap 2024-03-25T19:40:45.946 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.946+0000 7f880ec7b640 10 monclient: monmap: 2024-03-25T19:40:45.946 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:45.946 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:45.947 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:40:45.947633+0000 2024-03-25T19:40:45.947 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:40:45.947633+0000 2024-03-25T19:40:45.947 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:45.947 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:45.947 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:45.947 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:40:45.947 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:40:45.947 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:45.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.947+0000 7f880ec7b640 1 Processor -- start 2024-03-25T19:40:45.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.947+0000 7f880ec7b640 1 -- start start 2024-03-25T19:40:45.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.947+0000 7f880ec7b640 10 monclient: init 2024-03-25T19:40:45.948 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 10 monclient(hunting): picked mon.noname-a con 0x7f88080b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 10 monclient(hunting): picked mon.noname-b con 0x7f88080b1730 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 10 monclient(hunting): picked mon.noname-c con 0x7f880814be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88080af940 con 0x7f88080b4d60 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88080b0a70 con 0x7f880814be90 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88080b0d50 con 0x7f88080b1730 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.948+0000 7f880ec7b640 10 monclient(hunting): authenticate will time out at 6662.291897s 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.949+0000 7f8807fff640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f88080b1730 0x7f88080b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:56844/0 (socket says 172.21.15.143:56844) 2024-03-25T19:40:45.949 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:45.949+0000 7f8807fff640 1 -- 172.21.15.143:0/3673019783 learned_addr learned my addr 172.21.15.143:0/3673019783 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:46.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.026+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348688706 0 0) 0x7f88080af940 con 0x7f88080b4d60 2024-03-25T19:40:46.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.026+0000 7f88077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.026+0000 7f88077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.026+0000 7f88077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.026+0000 7f88077fe640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:40:46.026 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.026+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8003520 con 0x7f88080b4d60 2024-03-25T19:40:46.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.035+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079060399 0 0) 0x7f88080b0d50 con 0x7f88080b1730 2024-03-25T19:40:46.034 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.035+0000 7f88077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.035+0000 7f88077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.035+0000 7f88077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.035+0000 7f88077fe640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:40:46.035 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.035+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88080af940 con 0x7f88080b1730 2024-03-25T19:40:46.042 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 174164166 0 0) 0x7f87f8003520 con 0x7f88080b4d60 2024-03-25T19:40:46.042 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 >> v1:172.21.15.143:6790/0 conn(0x7f88080b1730 legacy=0x7f88080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.042 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 >> v1:172.21.15.203:6789/0 conn(0x7f880814be90 legacy=0x7f880814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.042 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:46.042 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 1 monclient: found mon.noname-a 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88080b1030 con 0x7f88080b4d60 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f87fc001690 con 0x7f88080b4d60 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f88077fe640 10 monclient: dump: 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.043 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.044 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:46.044 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:46.044 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:46.044 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:46.044 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:46.044 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f880ec7b640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:40:46.044 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.042+0000 7f880ec7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:46.058 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.059+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87fc0017f0 con 0x7f88080b4d60 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.059+0000 7f88077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.059+0000 7f88077fe640 1 -- 172.21.15.143:0/3673019783 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f87fc001c00 con 0x7f88080b4d60 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.059+0000 7f88077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.059+0000 7f88077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.059+0000 7f88077fe640 10 monclient: dump: 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:46.059 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:46.060 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:46.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.059+0000 7f880ec7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:46.060 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.059+0000 7f880ec7b640 4 set_mon_vals no callback set 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.060+0000 7f880ec7b640 10 monclient: shutdown 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.060+0000 7f880ec7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.060+0000 7f880ec7b640 1 -- 172.21.15.143:0/3673019783 >> v1:172.21.15.143:6789/0 conn(0x7f88080b4d60 legacy=0x7f880813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.060+0000 7f880ec7b640 1 -- 172.21.15.143:0/3673019783 shutdown_connections 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.060+0000 7f880ec7b640 1 -- 172.21.15.143:0/3673019783 >> 172.21.15.143:0/3673019783 conn(0x7f88080617a0 msgr2=0x7f8808061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.060+0000 7f880ec7b640 1 -- 172.21.15.143:0/3673019783 shutdown_connections 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.060+0000 7f880ec7b640 1 -- 172.21.15.143:0/3673019783 wait complete. 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 10 monclient: build_initial_monmap 2024-03-25T19:40:46.061 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 10 monclient: monmap: 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 1 librados: starting msgr at 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 1 librados: starting objecter 2024-03-25T19:40:46.062 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 1 Processor -- start 2024-03-25T19:40:46.063 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 1 -- start start 2024-03-25T19:40:46.063 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 1 librados: setting wanted keys 2024-03-25T19:40:46.063 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 1 librados: calling monclient init 2024-03-25T19:40:46.063 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.061+0000 7f880ec7b640 10 monclient: init 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 10 monclient(hunting): picked mon.noname-b con 0x7f88080b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 10 monclient(hunting): picked mon.noname-a con 0x7f88080b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 10 monclient(hunting): picked mon.noname-c con 0x7f880814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8808138210 con 0x7f88080b4d60 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88081375a0 con 0x7f88080b1730 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8808138440 con 0x7f880814be90 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:46.064 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.062+0000 7f880ec7b640 10 monclient(hunting): authenticate will time out at 6662.406593s 2024-03-25T19:40:46.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.068+0000 7f880d218640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f880814be90 0x7f88081e3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:56880/0 (socket says 172.21.15.143:56880) 2024-03-25T19:40:46.067 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.068+0000 7f880d218640 1 -- 172.21.15.143:0/110027150 learned_addr learned my addr 172.21.15.143:0/110027150 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:46.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.068+0000 7f88057fa640 10 client.?.objecter ms_handle_connect 0x7f880814be90 2024-03-25T19:40:46.068 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.068+0000 7f88057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.073+0000 7f88057fa640 10 client.?.objecter ms_handle_connect 0x7f88080b4d60 2024-03-25T19:40:46.073 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.073+0000 7f88057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.082+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890963089 0 0) 0x7f8808138440 con 0x7f880814be90 2024-03-25T19:40:46.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.082+0000 7f88057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.082+0000 7f88057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.082+0000 7f88057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.082+0000 7f88057fa640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:40:46.081 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.082+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87ec004eb0 con 0x7f880814be90 2024-03-25T19:40:46.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.087+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278689834 0 0) 0x7f8808138210 con 0x7f88080b4d60 2024-03-25T19:40:46.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.087+0000 7f88057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.087+0000 7f88057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.087+0000 7f88057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.087+0000 7f88057fa640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:40:46.087 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.087+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8808138440 con 0x7f88080b4d60 2024-03-25T19:40:46.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.091+0000 7f88057fa640 10 client.?.objecter ms_handle_connect 0x7f88080b1730 2024-03-25T19:40:46.090 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.091+0000 7f88057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:46.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.102+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891909517 0 0) 0x7f88081375a0 con 0x7f88080b1730 2024-03-25T19:40:46.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.102+0000 7f88057fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:46.101 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.102+0000 7f88057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:46.102 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.102+0000 7f88057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:46.102 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.102+0000 7f88057fa640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:40:46.102 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.102+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8808138210 con 0x7f88080b1730 2024-03-25T19:40:46.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.110+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 160211016 0 0) 0x7f8808138440 con 0x7f88080b4d60 2024-03-25T19:40:46.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.110+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88081375a0 con 0x7f88080b4d60 2024-03-25T19:40:46.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.110+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f87f4001930 con 0x7f88080b4d60 2024-03-25T19:40:46.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.110+0000 7f88057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.110+0000 7f88057fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:46.110 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.110+0000 7f88057fa640 10 monclient(hunting): dump: 2024-03-25T19:40:46.110 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:46.110 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:46.111 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.110+0000 7f88057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:46.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.117+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3147950341 0 0) 0x7f8808138210 con 0x7f88080b1730 2024-03-25T19:40:46.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.117+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8808138440 con 0x7f88080b1730 2024-03-25T19:40:46.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.117+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f87fc001b00 con 0x7f88080b1730 2024-03-25T19:40:46.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.117+0000 7f88057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.117+0000 7f88057fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:46.117 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.117+0000 7f88057fa640 10 monclient(hunting): dump: 2024-03-25T19:40:46.117 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:46.118 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.126+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3984468024 0 0) 0x7f87ec004eb0 con 0x7f880814be90 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.126+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8808138210 con 0x7f880814be90 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.126+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f8800001c30 con 0x7f880814be90 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.126+0000 7f88057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.126+0000 7f88057fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.126+0000 7f88057fa640 10 monclient(hunting): dump: 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.126 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:46.127 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:46.127 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:46.127 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:46.127 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:46.130 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4215666680 0 0) 0x7f88081375a0 con 0x7f88080b4d60 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 >> v1:172.21.15.143:6790/0 conn(0x7f880814be90 legacy=0x7f88081e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 >> v1:172.21.15.203:6789/0 conn(0x7f88080b1730 legacy=0x7f8808139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 1 monclient: found mon.a 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88081364b0 con 0x7f88080b4d60 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f88057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 10 monclient: _renew_subs 2024-03-25T19:40:46.131 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:46.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880813b470 con 0x7f88080b4d60 2024-03-25T19:40:46.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:46.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 10 monclient: _renew_subs 2024-03-25T19:40:46.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:46.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8808136230 con 0x7f88080b4d60 2024-03-25T19:40:46.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.130+0000 7f880ec7b640 1 librados: init done 2024-03-25T19:40:46.132 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.131+0000 7f880ec7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.131+0000 7f880ec7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.131+0000 7f880ec7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:46.133 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.131+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88080627b0 con 0x7f88080b4d60 2024-03-25T19:40:46.450 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.451+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87f4001d40 con 0x7f88080b4d60 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.451+0000 7f88057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.451+0000 7f8806ffd640 4 set_mon_vals no callback set 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.451+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f87f4004a40 con 0x7f88080b4d60 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.451+0000 7f88057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.451+0000 7f88057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.451+0000 7f88057fa640 10 monclient: dump: 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:46.451 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:46.452 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:46.452 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:46.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.452+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f87f4004c60 con 0x7f88080b4d60 2024-03-25T19:40:46.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.452+0000 7f880ca17640 10 monclient: get_auth_request con 0x7f87ec06e0f0 auth_method 0 2024-03-25T19:40:46.452 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.453+0000 7f88057fa640 10 client.4713.objecter ms_handle_connect 0x7f87ec06e0f0 2024-03-25T19:40:46.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.459+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7f87f40547b0 con 0x7f88080b4d60 2024-03-25T19:40:46.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.459+0000 7f88057fa640 10 client.4713.objecter ms_dispatch 0x7f8808000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:46.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.459+0000 7f88057fa640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:46.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.459+0000 7f88057fa640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:46.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.459+0000 7f88057fa640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T19:40:46.459 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.459+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87f40502c0 con 0x7f88080b4d60 2024-03-25T19:40:46.460 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.459+0000 7f88057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:46.460 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.459+0000 7f88057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:46.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.707+0000 7f880ec7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:46.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.707+0000 7f880ec7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:46.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.707+0000 7f880ec7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:46.707 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.707+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8808069f40 con 0x7f88080b4d60 2024-03-25T19:40:46.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.734+0000 7f88057fa640 1 -- 172.21.15.143:0/110027150 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7f87f40521f0 con 0x7f88080b4d60 2024-03-25T19:40:46.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.734+0000 7f88057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:46.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.734+0000 7f88057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:46.734 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:40:46.734 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 10 librados: watch_flush enter 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 10 librados: watch_flush exit 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 10 client.4713.objecter successfully canceled tick 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 >> v1:172.21.15.143:6800/1292282666 conn(0x7f87ec06e0f0 legacy=0x7f87ec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 10 monclient: shutdown 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 >> v1:172.21.15.143:6789/0 conn(0x7f88080b4d60 legacy=0x7f880812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:46.737 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 shutdown_connections 2024-03-25T19:40:46.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 >> 172.21.15.143:0/110027150 conn(0x7f88080617a0 msgr2=0x7f880813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:46.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 shutdown_connections 2024-03-25T19:40:46.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 1 -- 172.21.15.143:0/110027150 wait complete. 2024-03-25T19:40:46.738 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:46.737+0000 7f880ec7b640 1 librados: shutdown 2024-03-25T19:40:46.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:53.555 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:40:53.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.665+0000 7f6d8dc17640 10 monclient: get_monmap_and_config 2024-03-25T19:40:53.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.665+0000 7f6d8dc17640 10 monclient: build_initial_monmap 2024-03-25T19:40:53.665 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.665+0000 7f6d8dc17640 10 monclient: monmap: 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:40:53.667025+0000 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:40:53.667025+0000 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:40:53.666 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.665+0000 7f6d8dc17640 1 Processor -- start 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.665+0000 7f6d8dc17640 1 -- start start 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.665+0000 7f6d8dc17640 10 monclient: init 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 10 monclient(hunting): picked mon.noname-a con 0x7f6d880b4d60 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 10 monclient(hunting): picked mon.noname-c con 0x7f6d880b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 10 monclient(hunting): picked mon.noname-b con 0x7f6d8814be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d880af940 con 0x7f6d880b4d60 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d880b0a70 con 0x7f6d880b1730 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d880b0d50 con 0x7f6d8814be90 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.666+0000 7f6d8dc17640 10 monclient(hunting): authenticate will time out at 6670.011271s 2024-03-25T19:40:53.668 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.667+0000 7f6d86ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6d880b1730 0x7f6d880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:34150/0 (socket says 172.21.15.143:34150) 2024-03-25T19:40:53.669 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.667+0000 7f6d86ffd640 1 -- 172.21.15.143:0/3225601941 learned_addr learned my addr 172.21.15.143:0/3225601941 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:53.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.734+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190577896 0 0) 0x7f6d880af940 con 0x7f6d880b4d60 2024-03-25T19:40:53.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.734+0000 7f6d867fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.734 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.734+0000 7f6d867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.734+0000 7f6d867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.734+0000 7f6d867fc640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:40:53.735 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.734+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d70003520 con 0x7f6d880b4d60 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752038617 0 0) 0x7f6d70003520 con 0x7f6d880b4d60 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 >> v1:172.21.15.143:6790/0 conn(0x7f6d8814be90 legacy=0x7f6d8814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 >> v1:172.21.15.203:6789/0 conn(0x7f6d880b1730 legacy=0x7f6d880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 1 monclient: found mon.noname-a 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d880b1030 con 0x7f6d880b4d60 2024-03-25T19:40:53.746 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d78001690 con 0x7f6d880b4d60 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d867fc640 10 monclient: dump: 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:53.747 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:53.748 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:53.748 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:53.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d8dc17640 5 monclient: authenticate success, global_id 4716 2024-03-25T19:40:53.748 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.745+0000 7f6d8dc17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:53.760 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.760+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d780017f0 con 0x7f6d880b4d60 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.760+0000 7f6d867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.760+0000 7f6d867fc640 1 -- 172.21.15.143:0/3225601941 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d78001c00 con 0x7f6d880b4d60 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.760+0000 7f6d867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.760+0000 7f6d867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.760+0000 7f6d867fc640 10 monclient: dump: 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:53.761 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:53.762 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:53.762 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:53.762 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:53.762 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.760+0000 7f6d8dc17640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:53.762 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.760+0000 7f6d8dc17640 4 set_mon_vals no callback set 2024-03-25T19:40:53.763 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.761+0000 7f6d8dc17640 10 monclient: shutdown 2024-03-25T19:40:53.763 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.761+0000 7f6d8dc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:53.763 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.761+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3225601941 >> v1:172.21.15.143:6789/0 conn(0x7f6d880b4d60 legacy=0x7f6d8813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.761+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3225601941 shutdown_connections 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.761+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3225601941 >> 172.21.15.143:0/3225601941 conn(0x7f6d880617a0 msgr2=0x7f6d88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.761+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3225601941 shutdown_connections 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.761+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3225601941 wait complete. 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 10 monclient: build_initial_monmap 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 10 monclient: monmap: 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:40:53.764 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 1 librados: starting msgr at 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 1 librados: starting objecter 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 1 Processor -- start 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 1 -- start start 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 1 librados: setting wanted keys 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 1 librados: calling monclient init 2024-03-25T19:40:53.765 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.762+0000 7f6d8dc17640 10 monclient: init 2024-03-25T19:40:53.766 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:53.766 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:53.766 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 10 monclient(hunting): picked mon.noname-b con 0x7f6d880b1730 addr v1:172.21.15.203:6789/0 2024-03-25T19:40:53.767 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 10 monclient(hunting): picked mon.noname-c con 0x7f6d880b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:40:53.767 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 10 monclient(hunting): picked mon.noname-a con 0x7f6d8814be90 addr v1:172.21.15.143:6789/0 2024-03-25T19:40:53.767 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8813a940 con 0x7f6d8814be90 2024-03-25T19:40:53.767 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8813ab90 con 0x7f6d880b1730 2024-03-25T19:40:53.767 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d881d7840 con 0x7f6d880b4d60 2024-03-25T19:40:53.767 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:53.767 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.763+0000 7f6d8dc17640 10 monclient(hunting): authenticate will time out at 6670.108509s 2024-03-25T19:40:53.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.775+0000 7f6d87fff640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f6d8814be90 0x7f6d881e3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:33522/0 (socket says 172.21.15.143:33522) 2024-03-25T19:40:53.776 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.775+0000 7f6d87fff640 1 -- 172.21.15.143:0/3512023955 learned_addr learned my addr 172.21.15.143:0/3512023955 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:40:53.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.776+0000 7f6d6ffff640 10 client.?.objecter ms_handle_connect 0x7f6d8814be90 2024-03-25T19:40:53.777 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.776+0000 7f6d6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.786+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127693628 0 0) 0x7f6d8813a940 con 0x7f6d8814be90 2024-03-25T19:40:53.786 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.786+0000 7f6d6ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.786+0000 7f6d6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.786+0000 7f6d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.786+0000 7f6d6ffff640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:40:53.787 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.786+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d74004eb0 con 0x7f6d8814be90 2024-03-25T19:40:53.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.794+0000 7f6d6ffff640 10 client.?.objecter ms_handle_connect 0x7f6d880b4d60 2024-03-25T19:40:53.794 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.794+0000 7f6d6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.800+0000 7f6d6ffff640 10 client.?.objecter ms_handle_connect 0x7f6d880b1730 2024-03-25T19:40:53.801 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.800+0000 7f6d6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:53.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.810+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213941421 0 0) 0x7f6d881d7840 con 0x7f6d880b4d60 2024-03-25T19:40:53.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.810+0000 7f6d6ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.810+0000 7f6d6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.810+0000 7f6d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.810+0000 7f6d6ffff640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:40:53.810 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.810+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8813a940 con 0x7f6d880b4d60 2024-03-25T19:40:53.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.820+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246924955 0 0) 0x7f6d8813ab90 con 0x7f6d880b1730 2024-03-25T19:40:53.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.820+0000 7f6d6ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:40:53.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.820+0000 7f6d6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:53.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.820+0000 7f6d6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:53.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.820+0000 7f6d6ffff640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:40:53.820 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.820+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d881d7840 con 0x7f6d880b1730 2024-03-25T19:40:53.823 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.823+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1725528231 0 0) 0x7f6d8813a940 con 0x7f6d880b4d60 2024-03-25T19:40:53.823 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.823+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8813ab90 con 0x7f6d880b4d60 2024-03-25T19:40:53.823 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.823+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d7c0018a0 con 0x7f6d880b4d60 2024-03-25T19:40:53.823 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.823+0000 7f6d6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.823 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.823+0000 7f6d6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:53.823 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.823+0000 7f6d6ffff640 10 monclient(hunting): dump: 2024-03-25T19:40:53.823 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:53.824 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.823+0000 7f6d6ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.824+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3079853378 0 0) 0x7f6d74004eb0 con 0x7f6d8814be90 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.824+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8813a940 con 0x7f6d8814be90 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.824+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d80001c30 con 0x7f6d8814be90 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.824+0000 7f6d6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.824+0000 7f6d6ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.824+0000 7f6d6ffff640 10 monclient(hunting): dump: 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:53.825 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:53.826 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:53.826 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:53.826 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:53.826 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:53.826 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:53.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1258962488 0 0) 0x7f6d8813a940 con 0x7f6d8814be90 2024-03-25T19:40:53.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 >> v1:172.21.15.143:6790/0 conn(0x7f6d880b4d60 legacy=0x7f6d881d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 >> v1:172.21.15.203:6789/0 conn(0x7f6d880b1730 legacy=0x7f6d8813bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:53.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:40:53.830 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 1 monclient: found mon.a 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8813c240 con 0x7f6d8814be90 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 10 monclient: _finish_auth 0 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 5 monclient: authenticate success, global_id 4719 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 10 monclient: _renew_subs 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d88139ce0 con 0x7f6d8814be90 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 10 client.4719.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 10 monclient: _renew_subs 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d881d8650 con 0x7f6d8814be90 2024-03-25T19:40:53.831 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.830+0000 7f6d8dc17640 1 librados: init done 2024-03-25T19:40:53.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.831+0000 7f6d8dc17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.831+0000 7f6d8dc17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:53.832 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.831+0000 7f6d8dc17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:53.833 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:53.831+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d881401f0 con 0x7f6d8814be90 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.139+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d80003990 con 0x7f6d8814be90 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.139+0000 7f6d6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.139+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f6d80001d90 con 0x7f6d8814be90 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.139+0000 7f6d6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.139+0000 7f6d6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.139+0000 7f6d6ffff640 10 monclient: dump: 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:54.139 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:40:54.140 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:54.140 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:40:54.140 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:40:54.140 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:40:54.140 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:40:54.140 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:40:54.140 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.139+0000 7f6d85ffb640 4 set_mon_vals no callback set 2024-03-25T19:40:54.141 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.140+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f6d80003de0 con 0x7f6d8814be90 2024-03-25T19:40:54.141 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.140+0000 7f6d877fe640 10 monclient: get_auth_request con 0x7f6d7406dfb0 auth_method 0 2024-03-25T19:40:54.141 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.140+0000 7f6d6ffff640 10 client.4719.objecter ms_handle_connect 0x7f6d7406dfb0 2024-03-25T19:40:54.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.147+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7f6d80054940 con 0x7f6d8814be90 2024-03-25T19:40:54.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.147+0000 7f6d6ffff640 10 client.4719.objecter ms_dispatch 0x7f6d88000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:40:54.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.147+0000 7f6d6ffff640 3 client.4719.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:40:54.147 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.147+0000 7f6d6ffff640 3 client.4719.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:40:54.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.148+0000 7f6d6ffff640 20 client.4719.objecter dump_active .. 0 homeless 2024-03-25T19:40:54.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.148+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d80050450 con 0x7f6d8814be90 2024-03-25T19:40:54.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.148+0000 7f6d6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:54.148 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.148+0000 7f6d6ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:54.410 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.410+0000 7f6d8dc17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:54.410 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.410+0000 7f6d8dc17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:54.410 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.410+0000 7f6d8dc17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:40:54.410 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.410+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d880627b0 con 0x7f6d8814be90 2024-03-25T19:40:54.443 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.443+0000 7f6d6ffff640 1 -- 172.21.15.143:0/3512023955 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7f6d80052380 con 0x7f6d8814be90 2024-03-25T19:40:54.443 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.443+0000 7f6d6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:40:54.443 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.443+0000 7f6d6ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:54.443 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:40:54.443 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:40:54.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 10 librados: watch_flush enter 2024-03-25T19:40:54.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 10 librados: watch_flush exit 2024-03-25T19:40:54.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 20 client.4719.objecter shutdown clearing up homeless session... 2024-03-25T19:40:54.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 10 client.4719.objecter successfully canceled tick 2024-03-25T19:40:54.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 >> v1:172.21.15.143:6800/1292282666 conn(0x7f6d7406dfb0 legacy=0x7f6d74070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 10 monclient: shutdown 2024-03-25T19:40:54.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:54.446 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 >> v1:172.21.15.143:6789/0 conn(0x7f6d8814be90 legacy=0x7f6d881e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 shutdown_connections 2024-03-25T19:40:54.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 >> 172.21.15.143:0/3512023955 conn(0x7f6d880617a0 msgr2=0x7f6d88137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:54.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 shutdown_connections 2024-03-25T19:40:54.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 1 -- 172.21.15.143:0/3512023955 wait complete. 2024-03-25T19:40:54.447 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:40:54.446+0000 7f6d8dc17640 1 librados: shutdown 2024-03-25T19:40:54.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:40:58.006 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:24.570 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.016 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:24.581 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.022 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:24.587 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.023 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:24.559 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.026 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:24.591 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.027 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:24.563 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.047 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:24.583 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.047 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:24.613 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:58.100 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:24.665 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:01.264 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:01.378 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.378+0000 7f14dac57640 10 monclient: get_monmap_and_config 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.378+0000 7f14dac57640 10 monclient: build_initial_monmap 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.378+0000 7f14dac57640 10 monclient: monmap: 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:41:01.379432+0000 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:41:01.379432+0000 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.379+0000 7f14dac57640 1 Processor -- start 2024-03-25T19:41:01.379 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.379+0000 7f14dac57640 1 -- start start 2024-03-25T19:41:01.380 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.379+0000 7f14dac57640 10 monclient: init 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.379+0000 7f14dac57640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.379+0000 7f14dac57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.379+0000 7f14dac57640 10 monclient(hunting): picked mon.noname-b con 0x7f14d40a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.379+0000 7f14dac57640 10 monclient(hunting): picked mon.noname-c con 0x7f14d413f710 addr v1:172.21.15.203:6789/0 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.380+0000 7f14dac57640 10 monclient(hunting): picked mon.noname-a con 0x7f14d41430c0 addr v1:172.21.15.143:6789/0 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.380+0000 7f14dac57640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d4142520 con 0x7f14d41430c0 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.380+0000 7f14dac57640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d40b28a0 con 0x7f14d413f710 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.380+0000 7f14dac57640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d40b2a30 con 0x7f14d40a98f0 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.380+0000 7f14dac57640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:01.381 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.380+0000 7f14dac57640 10 monclient(hunting): authenticate will time out at 6677.723799s 2024-03-25T19:41:01.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.384+0000 7f14d3fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f14d413f710 0x7f14d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:41698/0 (socket says 172.21.15.143:41698) 2024-03-25T19:41:01.383 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.384+0000 7f14d3fff640 1 -- 172.21.15.143:0/1445604249 learned_addr learned my addr 172.21.15.143:0/1445604249 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:41:01.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.393+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825739125 0 0) 0x7f14d40b28a0 con 0x7f14d413f710 2024-03-25T19:41:01.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.393+0000 7f14d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.393+0000 7f14d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.393+0000 7f14d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.393+0000 7f14d37fe640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:41:01.393 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.393+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c4003520 con 0x7f14d413f710 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3662511737 0 0) 0x7f14c4003520 con 0x7f14d413f710 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 >> v1:172.21.15.143:6790/0 conn(0x7f14d40a98f0 legacy=0x7f14d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 >> v1:172.21.15.143:6789/0 conn(0x7f14d41430c0 legacy=0x7f14d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 1 monclient: found mon.noname-c 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d40b44c0 con 0x7f14d413f710 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:01.412 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f14c0001690 con 0x7f14d413f710 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14dac57640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14dac57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14d37fe640 10 monclient: dump: 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:01.413 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:01.414 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:01.414 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:01.414 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.412+0000 7f14dac57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.428+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14c0001860 con 0x7f14d413f710 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.428+0000 7f14d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.428+0000 7f14d37fe640 1 -- 172.21.15.143:0/1445604249 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f14c0001cb0 con 0x7f14d413f710 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.428+0000 7f14dac57640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.428+0000 7f14dac57640 4 set_mon_vals no callback set 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.428+0000 7f14d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.428+0000 7f14d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.428+0000 7f14d37fe640 10 monclient: dump: 2024-03-25T19:41:01.428 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:01.429 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:01.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.429+0000 7f14dac57640 10 monclient: shutdown 2024-03-25T19:41:01.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.429+0000 7f14dac57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:01.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.429+0000 7f14dac57640 1 -- 172.21.15.143:0/1445604249 >> v1:172.21.15.203:6789/0 conn(0x7f14d413f710 legacy=0x7f14d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.429+0000 7f14dac57640 1 -- 172.21.15.143:0/1445604249 shutdown_connections 2024-03-25T19:41:01.431 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.429+0000 7f14dac57640 1 -- 172.21.15.143:0/1445604249 >> 172.21.15.143:0/1445604249 conn(0x7f14d40b7ea0 msgr2=0x7f14d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.429+0000 7f14dac57640 1 -- 172.21.15.143:0/1445604249 shutdown_connections 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.429+0000 7f14dac57640 1 -- 172.21.15.143:0/1445604249 wait complete. 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 10 monclient: build_initial_monmap 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 10 monclient: monmap: 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:01.432 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 1 librados: starting msgr at 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 1 librados: starting objecter 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 1 Processor -- start 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 1 -- start start 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 1 librados: setting wanted keys 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 1 librados: calling monclient init 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.430+0000 7f14dac57640 10 monclient: init 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 10 monclient(hunting): picked mon.noname-b con 0x7f14d40a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 10 monclient(hunting): picked mon.noname-a con 0x7f14d413f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 10 monclient(hunting): picked mon.noname-c con 0x7f14d41430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d4139120 con 0x7f14d413f710 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d4139400 con 0x7f14d40a98f0 2024-03-25T19:41:01.433 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d4138300 con 0x7f14d41430c0 2024-03-25T19:41:01.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:01.434 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.431+0000 7f14dac57640 10 monclient(hunting): authenticate will time out at 6677.775341s 2024-03-25T19:41:01.438 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.439+0000 7f14d89f3640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f14d40a98f0 0x7f14d4137bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.143:41714/0 (socket says 172.21.15.143:41714) 2024-03-25T19:41:01.439 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.439+0000 7f14d89f3640 1 -- 172.21.15.143:0/1666663829 learned_addr learned my addr 172.21.15.143:0/1666663829 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:41:01.440 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.441+0000 7f14d17fa640 10 client.?.objecter ms_handle_connect 0x7f14d40a98f0 2024-03-25T19:41:01.441 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.441+0000 7f14d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.449+0000 7f14d17fa640 10 client.?.objecter ms_handle_connect 0x7f14d41430c0 2024-03-25T19:41:01.449 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.449+0000 7f14d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.456+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239956001 0 0) 0x7f14d4139400 con 0x7f14d40a98f0 2024-03-25T19:41:01.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.456+0000 7f14d17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.456+0000 7f14d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.456+0000 7f14d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.456+0000 7f14d17fa640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:41:01.456 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.456+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14b8004eb0 con 0x7f14d40a98f0 2024-03-25T19:41:01.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.457+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871335097 0 0) 0x7f14d4138300 con 0x7f14d41430c0 2024-03-25T19:41:01.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.457+0000 7f14d17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.457+0000 7f14d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.457+0000 7f14d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.457+0000 7f14d17fa640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:41:01.457 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.457+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d4139400 con 0x7f14d41430c0 2024-03-25T19:41:01.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.463+0000 7f14d17fa640 10 client.?.objecter ms_handle_connect 0x7f14d413f710 2024-03-25T19:41:01.462 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.463+0000 7f14d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.476+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3753059652 0 0) 0x7f14d4139400 con 0x7f14d41430c0 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.476+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d4138300 con 0x7f14d41430c0 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.476+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f14cc001a90 con 0x7f14d41430c0 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.476+0000 7f14d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.476+0000 7f14d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.476+0000 7f14d17fa640 10 monclient(hunting): dump: 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:01.476 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:01.477 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:01.477 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:01.477 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:01.477 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.476+0000 7f14d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:01.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545880487 0 0) 0x7f14d4139120 con 0x7f14d413f710 2024-03-25T19:41:01.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:41:01.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d4139400 con 0x7f14d413f710 2024-03-25T19:41:01.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2489922977 0 0) 0x7f14b8004eb0 con 0x7f14d40a98f0 2024-03-25T19:41:01.481 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14d4139120 con 0x7f14d40a98f0 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f14c8001a30 con 0x7f14d40a98f0 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.480+0000 7f14d17fa640 10 monclient(hunting): dump: 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:01.482 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.512+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3104933 0 0) 0x7f14d4139400 con 0x7f14d413f710 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.512+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14b8004eb0 con 0x7f14d413f710 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.512+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f14c00019d0 con 0x7f14d413f710 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.512+0000 7f14d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.512+0000 7f14d17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.512+0000 7f14d17fa640 10 monclient(hunting): dump: 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:01.512 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:01.513 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:01.513 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:01.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 611051394 0 0) 0x7f14b8004eb0 con 0x7f14d413f710 2024-03-25T19:41:01.527 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 >> v1:172.21.15.143:6790/0 conn(0x7f14d41430c0 legacy=0x7f14d41db9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 >> v1:172.21.15.203:6789/0 conn(0x7f14d40a98f0 legacy=0x7f14d4137bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 1 monclient: found mon.a 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d4137660 con 0x7f14d413f710 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 5 monclient: authenticate success, global_id 4722 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 10 monclient: _renew_subs 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d4136d30 con 0x7f14d413f710 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 10 client.4722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:01.528 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 10 monclient: _renew_subs 2024-03-25T19:41:01.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:41:01.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d4139a00 con 0x7f14d413f710 2024-03-25T19:41:01.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.527+0000 7f14dac57640 1 librados: init done 2024-03-25T19:41:01.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.528+0000 7f14dac57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.528+0000 7f14dac57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.528+0000 7f14dac57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:41:01.529 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.528+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d40b3ef0 con 0x7f14d413f710 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.905+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14c0001210 con 0x7f14d413f710 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.905+0000 7f14d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.905+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f14c0004ae0 con 0x7f14d413f710 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.905+0000 7f14d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.905+0000 7f14d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.905+0000 7f14d17fa640 10 monclient: dump: 2024-03-25T19:41:01.905 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:01.906 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.905+0000 7f14d2ffd640 4 set_mon_vals no callback set 2024-03-25T19:41:01.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.906+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f14c001ab30 con 0x7f14d413f710 2024-03-25T19:41:01.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.906+0000 7f14d89f3640 10 monclient: get_auth_request con 0x7f14b806e0f0 auth_method 0 2024-03-25T19:41:01.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.907+0000 7f14d17fa640 10 client.4722.objecter ms_handle_connect 0x7f14b806e0f0 2024-03-25T19:41:01.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.913+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7f14c0054770 con 0x7f14d413f710 2024-03-25T19:41:01.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.913+0000 7f14d17fa640 10 client.4722.objecter ms_dispatch 0x7f14d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:41:01.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.913+0000 7f14d17fa640 3 client.4722.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:41:01.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.913+0000 7f14d17fa640 3 client.4722.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:41:01.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.913+0000 7f14d17fa640 20 client.4722.objecter dump_active .. 0 homeless 2024-03-25T19:41:01.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.913+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14c00500b0 con 0x7f14d413f710 2024-03-25T19:41:01.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.913+0000 7f14d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.913 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:01.913+0000 7f14d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:02.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.162+0000 7f14dac57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.162+0000 7f14dac57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.162 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.162+0000 7f14dac57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.143:6789/0 2024-03-25T19:41:02.163 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.162+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 --> v1:172.21.15.143:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14d40627b0 con 0x7f14d413f710 2024-03-25T19:41:02.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.210+0000 7f14d17fa640 1 -- 172.21.15.143:0/1666663829 <== mon.0 v1:172.21.15.143:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7f14c00521b0 con 0x7f14d413f710 2024-03-25T19:41:02.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.210+0000 7f14d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:02.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.210+0000 7f14d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:02.210 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:41:02.210 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.212+0000 7f14dac57640 10 librados: watch_flush enter 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.212+0000 7f14dac57640 10 librados: watch_flush exit 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.212+0000 7f14dac57640 20 client.4722.objecter shutdown clearing up homeless session... 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.212+0000 7f14dac57640 10 client.4722.objecter successfully canceled tick 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.212+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 >> v1:172.21.15.143:6800/1292282666 conn(0x7f14b806e0f0 legacy=0x7f14b80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.212+0000 7f14dac57640 10 monclient: shutdown 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.212+0000 7f14dac57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.212+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 >> v1:172.21.15.143:6789/0 conn(0x7f14d413f710 legacy=0x7f14d41db2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.213+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 shutdown_connections 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.213+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 >> 172.21.15.143:0/1666663829 conn(0x7f14d40b7ea0 msgr2=0x7f14d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.213+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 shutdown_connections 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.213+0000 7f14dac57640 1 -- 172.21.15.143:0/1666663829 wait complete. 2024-03-25T19:41:02.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:02.213+0000 7f14dac57640 1 librados: shutdown 2024-03-25T19:41:02.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:04.897 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:31.462 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:04.902 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:31.467 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:04.948 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:31.513 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:04.953 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:31.518 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:04.961 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:31.497 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:04.966 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:31.502 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:04.966 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:31.531 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:04.987 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:31.552 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:05.000 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:31.536 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:09.031 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.150+0000 7f76a6982640 10 monclient: get_monmap_and_config 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.150+0000 7f76a6982640 10 monclient: build_initial_monmap 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.150+0000 7f76a6982640 10 monclient: monmap: 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:41:09.151325+0000 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:41:09.151325+0000 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:41:09.150 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:41:09.151 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:41:09.151 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:09.151 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.151+0000 7f76a6982640 1 Processor -- start 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.151+0000 7f76a6982640 1 -- start start 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.151+0000 7f76a6982640 10 monclient: init 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 10 monclient(hunting): picked mon.noname-b con 0x7f76a00a98f0 addr v1:172.21.15.143:6790/0 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 10 monclient(hunting): picked mon.noname-a con 0x7f76a013f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 10 monclient(hunting): picked mon.noname-c con 0x7f76a01430c0 addr v1:172.21.15.203:6789/0 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a0142520 con 0x7f76a013f710 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a00b28a0 con 0x7f76a01430c0 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a00b2a30 con 0x7f76a00a98f0 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:09.152 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.152+0000 7f76a6982640 10 monclient(hunting): authenticate will time out at 6685.496028s 2024-03-25T19:41:09.166 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.166+0000 7f769f7fe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f76a013f710 0x7f76a01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:33718/0 (socket says 172.21.15.143:33718) 2024-03-25T19:41:09.166 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.166+0000 7f769f7fe640 1 -- 172.21.15.143:0/1705453530 learned_addr learned my addr 172.21.15.143:0/1705453530 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:41:09.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.182+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580342705 0 0) 0x7f76a00b28a0 con 0x7f76a01430c0 2024-03-25T19:41:09.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.182+0000 7f769effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.182+0000 7f769effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.182+0000 7f769effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.182+0000 7f769effd640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:41:09.182 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.182+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7688003520 con 0x7f76a01430c0 2024-03-25T19:41:09.192 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.192+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377575860 0 0) 0x7f76a0142520 con 0x7f76a013f710 2024-03-25T19:41:09.192 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.192+0000 7f769effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.192 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.192+0000 7f769effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.192 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.192+0000 7f769effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.192 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.192+0000 7f769effd640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:41:09.192 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.192+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a00b28a0 con 0x7f76a013f710 2024-03-25T19:41:09.200 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1488011732 0 0) 0x7f76a00b28a0 con 0x7f76a013f710 2024-03-25T19:41:09.200 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 >> v1:172.21.15.143:6790/0 conn(0x7f76a00a98f0 legacy=0x7f76a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.200 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 >> v1:172.21.15.203:6789/0 conn(0x7f76a01430c0 legacy=0x7f76a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 1 monclient: found mon.noname-a 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a00b44c0 con 0x7f76a013f710 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 10 monclient: _finish_auth 0 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7694001670 con 0x7f76a013f710 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.201 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f769effd640 10 monclient: dump: 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:09.202 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:09.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f76a6982640 5 monclient: authenticate success, global_id 4725 2024-03-25T19:41:09.203 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.200+0000 7f76a6982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:09.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.209+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76940017d0 con 0x7f76a013f710 2024-03-25T19:41:09.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.209+0000 7f769effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:09.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.209+0000 7f769effd640 1 -- 172.21.15.143:0/1705453530 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7694001ba0 con 0x7f76a013f710 2024-03-25T19:41:09.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.209+0000 7f769effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.209+0000 7f769effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:09.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.209+0000 7f769effd640 10 monclient: dump: 2024-03-25T19:41:09.209 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.209+0000 7f76a6982640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:09.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.209+0000 7f76a6982640 4 set_mon_vals no callback set 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.210+0000 7f76a6982640 10 monclient: shutdown 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.210+0000 7f76a6982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.210+0000 7f76a6982640 1 -- 172.21.15.143:0/1705453530 >> v1:172.21.15.143:6789/0 conn(0x7f76a013f710 legacy=0x7f76a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.210+0000 7f76a6982640 1 -- 172.21.15.143:0/1705453530 shutdown_connections 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.210+0000 7f76a6982640 1 -- 172.21.15.143:0/1705453530 >> 172.21.15.143:0/1705453530 conn(0x7f76a00b7ea0 msgr2=0x7f76a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.210+0000 7f76a6982640 1 -- 172.21.15.143:0/1705453530 shutdown_connections 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.210+0000 7f76a6982640 1 -- 172.21.15.143:0/1705453530 wait complete. 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 10 monclient: build_initial_monmap 2024-03-25T19:41:09.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 10 monclient: monmap: 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:41:09.214 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 1 librados: starting msgr at 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 1 librados: starting objecter 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 1 Processor -- start 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 1 -- start start 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 1 librados: setting wanted keys 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 1 librados: calling monclient init 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.211+0000 7f76a6982640 10 monclient: init 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 10 monclient(hunting): picked mon.noname-b con 0x7f76a00a98f0 addr v1:172.21.15.203:6789/0 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 10 monclient(hunting): picked mon.noname-a con 0x7f76a013f710 addr v1:172.21.15.143:6789/0 2024-03-25T19:41:09.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 10 monclient(hunting): picked mon.noname-c con 0x7f76a01430c0 addr v1:172.21.15.143:6790/0 2024-03-25T19:41:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a013a3f0 con 0x7f76a013f710 2024-03-25T19:41:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a013b6d0 con 0x7f76a00a98f0 2024-03-25T19:41:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a013b9b0 con 0x7f76a01430c0 2024-03-25T19:41:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.212+0000 7f76a6982640 10 monclient(hunting): authenticate will time out at 6685.556265s 2024-03-25T19:41:09.216 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.217+0000 7f76a4f1f640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f76a01430c0 0x7f76a00b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:55604/0 (socket says 172.21.15.143:55604) 2024-03-25T19:41:09.217 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.217+0000 7f76a4f1f640 1 -- 172.21.15.143:0/3153050487 learned_addr learned my addr 172.21.15.143:0/3153050487 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:41:09.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.220+0000 7f769cff9640 10 client.?.objecter ms_handle_connect 0x7f76a01430c0 2024-03-25T19:41:09.220 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.220+0000 7f769cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.222+0000 7f769cff9640 10 client.?.objecter ms_handle_connect 0x7f76a013f710 2024-03-25T19:41:09.222 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.222+0000 7f769cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.227 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.228+0000 7f769cff9640 10 client.?.objecter ms_handle_connect 0x7f76a00a98f0 2024-03-25T19:41:09.227 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.228+0000 7f769cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:09.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.234+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658698731 0 0) 0x7f76a013b9b0 con 0x7f76a01430c0 2024-03-25T19:41:09.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.234+0000 7f769cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.234+0000 7f769cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.234+0000 7f769cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.234+0000 7f769cff9640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:41:09.233 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.234+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c004eb0 con 0x7f76a01430c0 2024-03-25T19:41:09.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.240+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108883499 0 0) 0x7f76a013b6d0 con 0x7f76a00a98f0 2024-03-25T19:41:09.239 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.240+0000 7f769cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.240+0000 7f769cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.240+0000 7f769cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.240+0000 7f769cff9640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:41:09.240 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.240+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a013b9b0 con 0x7f76a00a98f0 2024-03-25T19:41:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.241+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722176484 0 0) 0x7f76a013a3f0 con 0x7f76a013f710 2024-03-25T19:41:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.241+0000 7f769cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.241+0000 7f769cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.241+0000 7f769cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.241+0000 7f769cff9640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:41:09.241 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.241+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a013b6d0 con 0x7f76a013f710 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.247+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 939968374 0 0) 0x7f768c004eb0 con 0x7f76a01430c0 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.247+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76a013a3f0 con 0x7f76a01430c0 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.247+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7698001c70 con 0x7f76a01430c0 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.247+0000 7f769cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.247+0000 7f769cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.247+0000 7f769cff9640 10 monclient(hunting): dump: 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.247 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.248 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.248 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:09.248 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:09.248 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:09.248 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:09.248 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:09.248 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.247+0000 7f769cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:09.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.256+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2347529251 0 0) 0x7f76a013b6d0 con 0x7f76a013f710 2024-03-25T19:41:09.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.256+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f768c004eb0 con 0x7f76a013f710 2024-03-25T19:41:09.255 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.256+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7694001530 con 0x7f76a013f710 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.256+0000 7f769cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.256+0000 7f769cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.256+0000 7f769cff9640 10 monclient(hunting): dump: 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:09.256 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:09.257 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:09.257 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:09.257 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:09.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.257+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3772660296 0 0) 0x7f76a013a3f0 con 0x7f76a01430c0 2024-03-25T19:41:09.258 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.257+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 >> v1:172.21.15.203:6789/0 conn(0x7f76a00a98f0 legacy=0x7f76a013bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.257+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 >> v1:172.21.15.143:6789/0 conn(0x7f76a013f710 legacy=0x7f76a012f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.257+0000 7f769cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.257+0000 7f769cff9640 1 monclient: found mon.c 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.257+0000 7f769cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.257+0000 7f769cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.257+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a0130020 con 0x7f76a01430c0 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f769cff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f769cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f769cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 5 monclient: authenticate success, global_id 4721 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 10 monclient: _renew_subs 2024-03-25T19:41:09.259 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a013a700 con 0x7f76a01430c0 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 10 client.4721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 10 monclient: _renew_subs 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a01e3690 con 0x7f76a01430c0 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.258+0000 7f76a6982640 1 librados: init done 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.259+0000 7f76a6982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.259+0000 7f76a6982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.259+0000 7f76a6982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:09.260 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.259+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a013a1d0 con 0x7f76a01430c0 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.576+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7698003990 con 0x7f76a01430c0 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.576+0000 7f769cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.576+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f7698003da0 con 0x7f76a01430c0 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.576+0000 7f769cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.576+0000 7f769cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.576+0000 7f769cff9640 10 monclient: dump: 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:09.576 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.577 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:09.577 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:09.577 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:09.577 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:09.577 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:09.577 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:09.577 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:09.577 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.576+0000 7f769e7fc640 4 set_mon_vals no callback set 2024-03-25T19:41:09.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.577+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f7698003fc0 con 0x7f76a01430c0 2024-03-25T19:41:09.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.578+0000 7f769ffff640 10 monclient: get_auth_request con 0x7f768c06df70 auth_method 0 2024-03-25T19:41:09.578 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.578+0000 7f769cff9640 10 client.4721.objecter ms_handle_connect 0x7f768c06df70 2024-03-25T19:41:09.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.583+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7f7698054a80 con 0x7f76a01430c0 2024-03-25T19:41:09.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.583+0000 7f769cff9640 10 client.4721.objecter ms_dispatch 0x7f76a0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:41:09.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.583+0000 7f769cff9640 3 client.4721.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:41:09.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.583+0000 7f769cff9640 3 client.4721.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:41:09.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.583+0000 7f769cff9640 20 client.4721.objecter dump_active .. 0 homeless 2024-03-25T19:41:09.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.583+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7698050590 con 0x7f76a01430c0 2024-03-25T19:41:09.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.583+0000 7f769cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:09.583 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.583+0000 7f769cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:09.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.849+0000 7f76a6982640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:09.848 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.849+0000 7f76a6982640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:09.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.849+0000 7f76a6982640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:09.849 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.849+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76a00b3ed0 con 0x7f76a01430c0 2024-03-25T19:41:09.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.876+0000 7f769cff9640 1 -- 172.21.15.143:0/3153050487 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7f76980524c0 con 0x7f76a01430c0 2024-03-25T19:41:09.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.876+0000 7f769cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:09.875 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.876+0000 7f769cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:09.876 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:41:09.876 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:09.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 10 librados: watch_flush enter 2024-03-25T19:41:09.878 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 10 librados: watch_flush exit 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 20 client.4721.objecter shutdown clearing up homeless session... 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 10 client.4721.objecter successfully canceled tick 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 >> v1:172.21.15.143:6800/1292282666 conn(0x7f768c06df70 legacy=0x7f768c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 10 monclient: shutdown 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 >> v1:172.21.15.143:6790/0 conn(0x7f76a01430c0 legacy=0x7f76a00b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 shutdown_connections 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.878+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 >> 172.21.15.143:0/3153050487 conn(0x7f76a00b7ea0 msgr2=0x7f76a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.879+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 shutdown_connections 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.879+0000 7f76a6982640 1 -- 172.21.15.143:0/3153050487 wait complete. 2024-03-25T19:41:09.879 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:09.879+0000 7f76a6982640 1 librados: shutdown 2024-03-25T19:41:09.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:16.698 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:16.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.813+0000 7f5deb763640 10 monclient: get_monmap_and_config 2024-03-25T19:41:16.813 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.813+0000 7f5deb763640 10 monclient: build_initial_monmap 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.813+0000 7f5deb763640 10 monclient: monmap: 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:41:16.814805+0000 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:41:16.814805+0000 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:41:16.814 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:16.816 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.814+0000 7f5deb763640 1 Processor -- start 2024-03-25T19:41:16.816 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.814+0000 7f5deb763640 1 -- start start 2024-03-25T19:41:16.816 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.814+0000 7f5deb763640 10 monclient: init 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 10 monclient(hunting): picked mon.noname-b con 0x7f5de40b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 10 monclient(hunting): picked mon.noname-a con 0x7f5de40b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 10 monclient(hunting): picked mon.noname-c con 0x7f5de414be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de40af940 con 0x7f5de40b1730 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de40b0a70 con 0x7f5de414be90 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de40b0d50 con 0x7f5de40b4d60 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:16.817 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.815+0000 7f5deb763640 10 monclient(hunting): authenticate will time out at 6693.159346s 2024-03-25T19:41:16.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.823+0000 7f5de8cfe640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7f5de40b1730 0x7f5de40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:51306/0 (socket says 172.21.15.143:51306) 2024-03-25T19:41:16.822 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.823+0000 7f5de8cfe640 1 -- 172.21.15.143:0/4019826613 learned_addr learned my addr 172.21.15.143:0/4019826613 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:41:16.834 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.835+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401955015 0 0) 0x7f5de40af940 con 0x7f5de40b1730 2024-03-25T19:41:16.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.835+0000 7f5ddbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:16.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.835+0000 7f5ddbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:16.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.835+0000 7f5ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:16.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.835+0000 7f5ddbfff640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:41:16.835 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.835+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd0003520 con 0x7f5de40b1730 2024-03-25T19:41:16.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135530450 0 0) 0x7f5de40b0d50 con 0x7f5de40b4d60 2024-03-25T19:41:16.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:16.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:16.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:16.854 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de40af940 con 0x7f5de40b4d60 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 <== mon.0 v1:172.21.15.143:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538523469 0 0) 0x7f5dd0003520 con 0x7f5de40b1730 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 >> v1:172.21.15.143:6790/0 conn(0x7f5de40b4d60 legacy=0x7f5de413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 >> v1:172.21.15.203:6789/0 conn(0x7f5de414be90 legacy=0x7f5de414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 1 monclient: found mon.noname-a 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.143:6789/0 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 --> v1:172.21.15.143:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de40b1030 con 0x7f5de40b1730 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 <== mon.0 v1:172.21.15.143:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5dd4001690 con 0x7f5de40b1730 2024-03-25T19:41:16.855 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5deb763640 5 monclient: authenticate success, global_id 4731 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5deb763640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5ddbfff640 10 monclient: dump: 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:16.856 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.854+0000 7f5deb763640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:16.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 <== mon.0 v1:172.21.15.143:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dd4001870 con 0x7f5de40b1730 2024-03-25T19:41:16.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5ddbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:16.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5ddbfff640 1 -- 172.21.15.143:0/4019826613 <== mon.0 v1:172.21.15.143:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5dd4001c80 con 0x7f5de40b1730 2024-03-25T19:41:16.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5deb763640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:16.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5deb763640 4 set_mon_vals no callback set 2024-03-25T19:41:16.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5ddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:16.859 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5ddbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5ddbfff640 10 monclient: dump: 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:16.860 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:16.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5deb763640 10 monclient: shutdown 2024-03-25T19:41:16.861 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5deb763640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.860+0000 7f5deb763640 1 -- 172.21.15.143:0/4019826613 >> v1:172.21.15.143:6789/0 conn(0x7f5de40b1730 legacy=0x7f5de40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.861+0000 7f5deb763640 1 -- 172.21.15.143:0/4019826613 shutdown_connections 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.861+0000 7f5deb763640 1 -- 172.21.15.143:0/4019826613 >> 172.21.15.143:0/4019826613 conn(0x7f5de40617a0 msgr2=0x7f5de4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.861+0000 7f5deb763640 1 -- 172.21.15.143:0/4019826613 shutdown_connections 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.861+0000 7f5deb763640 1 -- 172.21.15.143:0/4019826613 wait complete. 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.861+0000 7f5deb763640 10 monclient: build_initial_monmap 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.861+0000 7f5deb763640 10 monclient: monmap: 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:41:16.862 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.862+0000 7f5deb763640 1 librados: starting msgr at 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.862+0000 7f5deb763640 1 librados: starting objecter 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.862+0000 7f5deb763640 1 Processor -- start 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.862+0000 7f5deb763640 1 -- start start 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.862+0000 7f5deb763640 1 librados: setting wanted keys 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.862+0000 7f5deb763640 1 librados: calling monclient init 2024-03-25T19:41:16.863 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.862+0000 7f5deb763640 10 monclient: init 2024-03-25T19:41:16.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:16.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:16.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 10 monclient(hunting): picked mon.noname-a con 0x7f5de40b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:41:16.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 10 monclient(hunting): picked mon.noname-c con 0x7f5de40b4d60 addr v1:172.21.15.143:6790/0 2024-03-25T19:41:16.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 10 monclient(hunting): picked mon.noname-b con 0x7f5de414be90 addr v1:172.21.15.203:6789/0 2024-03-25T19:41:16.864 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de41daca0 con 0x7f5de40b1730 2024-03-25T19:41:16.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de41dbb00 con 0x7f5de414be90 2024-03-25T19:41:16.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de41daed0 con 0x7f5de40b4d60 2024-03-25T19:41:16.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:16.865 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.863+0000 7f5deb763640 10 monclient(hunting): authenticate will time out at 6693.206990s 2024-03-25T19:41:16.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.884+0000 7f5de8cfe640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7f5de40b4d60 0x7f5de41db3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:38252/0 (socket says 172.21.15.143:38252) 2024-03-25T19:41:16.883 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.884+0000 7f5de8cfe640 1 -- 172.21.15.143:0/1658663970 learned_addr learned my addr 172.21.15.143:0/1658663970 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:41:16.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.884+0000 7f5dd9ffb640 10 client.?.objecter ms_handle_connect 0x7f5de40b4d60 2024-03-25T19:41:16.884 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.884+0000 7f5dd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:16.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.890+0000 7f5dd9ffb640 10 client.?.objecter ms_handle_connect 0x7f5de414be90 2024-03-25T19:41:16.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.890+0000 7f5dd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:16.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.890+0000 7f5dd9ffb640 10 client.?.objecter ms_handle_connect 0x7f5de40b1730 2024-03-25T19:41:16.890 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.890+0000 7f5dd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:16.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.893+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850223086 0 0) 0x7f5de41daed0 con 0x7f5de40b4d60 2024-03-25T19:41:16.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.893+0000 7f5dd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:16.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.893+0000 7f5dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:16.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.893+0000 7f5dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:16.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.893+0000 7f5dd9ffb640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:41:16.893 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.893+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc8004eb0 con 0x7f5de40b4d60 2024-03-25T19:41:16.895 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.896+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320145727 0 0) 0x7f5de41dbb00 con 0x7f5de414be90 2024-03-25T19:41:16.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.896+0000 7f5dd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:16.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.896+0000 7f5dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:16.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.896+0000 7f5dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:16.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.896+0000 7f5dd9ffb640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:41:16.896 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.896+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de41daed0 con 0x7f5de414be90 2024-03-25T19:41:16.901 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.902+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1939580615 0 0) 0x7f5de41daed0 con 0x7f5de414be90 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.902+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5de41dbb00 con 0x7f5de414be90 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.902+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5ddc001b00 con 0x7f5de414be90 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.902+0000 7f5dd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.902+0000 7f5dd9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.902+0000 7f5dd9ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:16.902 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:16.903 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:16.903 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.902+0000 7f5dd9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.907+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277746643 0 0) 0x7f5de41daca0 con 0x7f5de40b1730 2024-03-25T19:41:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.908+0000 7f5dd9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.908+0000 7f5dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.908+0000 7f5dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.908+0000 7f5dd9ffb640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:41:16.907 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.908+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de41daed0 con 0x7f5de40b1730 2024-03-25T19:41:16.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.910+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2526120927 0 0) 0x7f5dc8004eb0 con 0x7f5de40b4d60 2024-03-25T19:41:16.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.910+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5de41daca0 con 0x7f5de40b4d60 2024-03-25T19:41:16.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.911+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5dd4001c40 con 0x7f5de40b4d60 2024-03-25T19:41:16.910 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.911+0000 7f5dd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.911+0000 7f5dd9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.911+0000 7f5dd9ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:16.911 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:16.915 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 434000971 0 0) 0x7f5de41dbb00 con 0x7f5de414be90 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 >> v1:172.21.15.143:6790/0 conn(0x7f5de40b4d60 legacy=0x7f5de41db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 >> v1:172.21.15.143:6789/0 conn(0x7f5de40b1730 legacy=0x7f5de41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 1 monclient: found mon.b 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de41389d0 con 0x7f5de414be90 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5dd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5deb763640 5 monclient: authenticate success, global_id 4741 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5deb763640 10 monclient: _renew_subs 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5deb763640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:41:16.916 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.915+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de4143090 con 0x7f5de414be90 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.916+0000 7f5deb763640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.916+0000 7f5deb763640 10 monclient: _renew_subs 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.916+0000 7f5deb763640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.916+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de4138080 con 0x7f5de414be90 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.916+0000 7f5deb763640 1 librados: init done 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.917+0000 7f5deb763640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.917+0000 7f5deb763640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.917+0000 7f5deb763640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:41:16.917 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:16.917+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de40627b0 con 0x7f5de414be90 2024-03-25T19:41:17.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.208+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ddc001fb0 con 0x7f5de414be90 2024-03-25T19:41:17.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.208+0000 7f5dd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:17.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.208+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7f5ddc003b10 con 0x7f5de414be90 2024-03-25T19:41:17.208 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.208+0000 7f5dd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.208+0000 7f5dd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.208+0000 7f5dd9ffb640 10 monclient: dump: 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:17.209 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.208+0000 7f5ddb7fe640 4 set_mon_vals no callback set 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.209+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7f5ddc003d30 con 0x7f5de414be90 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.210+0000 7f5de94ff640 10 monclient: get_auth_request con 0x7f5dc806dfb0 auth_method 0 2024-03-25T19:41:17.210 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.210+0000 7f5dd9ffb640 10 client.4741.objecter ms_handle_connect 0x7f5dc806dfb0 2024-03-25T19:41:17.213 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.214+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7f5ddc0536b0 con 0x7f5de414be90 2024-03-25T19:41:17.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.214+0000 7f5dd9ffb640 10 client.4741.objecter ms_dispatch 0x7f5de4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:41:17.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.214+0000 7f5dd9ffb640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:41:17.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.214+0000 7f5dd9ffb640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:41:17.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.214+0000 7f5dd9ffb640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T19:41:17.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.214+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ddc04f1c0 con 0x7f5de414be90 2024-03-25T19:41:17.214 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.214+0000 7f5dd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:17.215 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.214+0000 7f5dd9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:17.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.468+0000 7f5deb763640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:17.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.468+0000 7f5deb763640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:17.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.468+0000 7f5deb763640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-25T19:41:17.468 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.468+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5de4069f40 con 0x7f5de414be90 2024-03-25T19:41:17.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.625+0000 7f5dd9ffb640 1 -- 172.21.15.143:0/1658663970 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7f5ddc0510f0 con 0x7f5de414be90 2024-03-25T19:41:17.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.625+0000 7f5dd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:17.625 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.625+0000 7f5dd9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:17.626 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:41:17.626 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:17.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 10 librados: watch_flush enter 2024-03-25T19:41:17.628 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 10 librados: watch_flush exit 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 10 client.4741.objecter successfully canceled tick 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 >> v1:172.21.15.143:6800/1292282666 conn(0x7f5dc806dfb0 legacy=0x7f5dc8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 10 monclient: shutdown 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 >> v1:172.21.15.203:6789/0 conn(0x7f5de414be90 legacy=0x7f5de4142980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.628+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 shutdown_connections 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.629+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 >> 172.21.15.143:0/1658663970 conn(0x7f5de40617a0 msgr2=0x7f5de413c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.629+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 shutdown_connections 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.629+0000 7f5deb763640 1 -- 172.21.15.143:0/1658663970 wait complete. 2024-03-25T19:41:17.629 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:17.629+0000 7f5deb763640 1 librados: shutdown 2024-03-25T19:41:17.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:24.447 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:24.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.565+0000 7ffb69d3d640 10 monclient: get_monmap_and_config 2024-03-25T19:41:24.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.565+0000 7ffb69d3d640 10 monclient: build_initial_monmap 2024-03-25T19:41:24.565 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.565+0000 7ffb69d3d640 10 monclient: monmap: 2024-03-25T19:41:24.565 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:41:24.565 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:24.566 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:41:24.566528+0000 2024-03-25T19:41:24.566 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:41:24.566528+0000 2024-03-25T19:41:24.566 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:24.566 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:24.566 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:41:24.566 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-25T19:41:24.566 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-b 2024-03-25T19:41:24.566 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.566+0000 7ffb69d3d640 1 Processor -- start 2024-03-25T19:41:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.566+0000 7ffb69d3d640 1 -- start start 2024-03-25T19:41:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.566+0000 7ffb69d3d640 10 monclient: init 2024-03-25T19:41:24.567 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb640b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb640b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6414be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb640af940 con 0x7ffb640b1730 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb640b0a70 con 0x7ffb640b4d60 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb640b0d50 con 0x7ffb6414be90 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:24.568 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.567+0000 7ffb69d3d640 10 monclient(hunting): authenticate will time out at 6700.910900s 2024-03-25T19:41:24.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.580+0000 7ffb62ffd640 1 --1- >> v1:172.21.15.143:6789/0 conn(0x7ffb640b1730 0x7ffb640b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.143:6789/0 says I am v1:172.21.15.143:56284/0 (socket says 172.21.15.143:56284) 2024-03-25T19:41:24.580 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.580+0000 7ffb62ffd640 1 -- 172.21.15.143:0/1165957718 learned_addr learned my addr 172.21.15.143:0/1165957718 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:41:24.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.595+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 <== mon.0 v1:172.21.15.143:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141868235 0 0) 0x7ffb640af940 con 0x7ffb640b1730 2024-03-25T19:41:24.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.595+0000 7ffb627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.595+0000 7ffb627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.595+0000 7ffb627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.595+0000 7ffb627fc640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:41:24.595 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.595+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 --> v1:172.21.15.143:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb4c003520 con 0x7ffb640b1730 2024-03-25T19:41:24.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.596+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277312137 0 0) 0x7ffb640b0d50 con 0x7ffb6414be90 2024-03-25T19:41:24.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.596+0000 7ffb627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.596+0000 7ffb627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.596+0000 7ffb627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.596+0000 7ffb627fc640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:41:24.596 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.596+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb640af940 con 0x7ffb6414be90 2024-03-25T19:41:24.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.598+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620088964 0 0) 0x7ffb640b0a70 con 0x7ffb640b4d60 2024-03-25T19:41:24.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.598+0000 7ffb627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.598+0000 7ffb627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.598+0000 7ffb627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.598+0000 7ffb627fc640 10 monclient(hunting): my global_id is 4744 2024-03-25T19:41:24.598 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.598+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb640b0d50 con 0x7ffb640b4d60 2024-03-25T19:41:24.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3358230487 0 0) 0x7ffb640af940 con 0x7ffb6414be90 2024-03-25T19:41:24.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 >> v1:172.21.15.203:6789/0 conn(0x7ffb640b4d60 legacy=0x7ffb6413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 >> v1:172.21.15.143:6789/0 conn(0x7ffb640b1730 legacy=0x7ffb640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:24.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 1 monclient: found mon.noname-b 2024-03-25T19:41:24.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:24.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.143:6790/0 2024-03-25T19:41:24.608 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb640b1030 con 0x7ffb6414be90 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ffb5c0019f0 con 0x7ffb6414be90 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb69d3d640 5 monclient: authenticate success, global_id 4730 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb627fc640 10 monclient: dump: 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:24.609 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:24.610 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.610 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:24.610 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:24.610 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:24.610 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:24.610 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:24.610 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.608+0000 7ffb69d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.615+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 <== mon.2 v1:172.21.15.143:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb5c001b50 con 0x7ffb6414be90 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.615+0000 7ffb627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.615+0000 7ffb627fc640 1 -- 172.21.15.143:0/1165957718 <== mon.2 v1:172.21.15.143:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ffb5c001f60 con 0x7ffb6414be90 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.615+0000 7ffb69d3d640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.615+0000 7ffb69d3d640 4 set_mon_vals no callback set 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.615+0000 7ffb627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.615+0000 7ffb627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.615+0000 7ffb627fc640 10 monclient: dump: 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:24.615 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.616 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:24.616 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:24.616 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:24.616 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:24.616 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:24.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.616+0000 7ffb69d3d640 10 monclient: shutdown 2024-03-25T19:41:24.618 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.616+0000 7ffb69d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.616+0000 7ffb69d3d640 1 -- 172.21.15.143:0/1165957718 >> v1:172.21.15.143:6790/0 conn(0x7ffb6414be90 legacy=0x7ffb6414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.616+0000 7ffb69d3d640 1 -- 172.21.15.143:0/1165957718 shutdown_connections 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.616+0000 7ffb69d3d640 1 -- 172.21.15.143:0/1165957718 >> 172.21.15.143:0/1165957718 conn(0x7ffb640617a0 msgr2=0x7ffb64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.616+0000 7ffb69d3d640 1 -- 172.21.15.143:0/1165957718 shutdown_connections 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.616+0000 7ffb69d3d640 1 -- 172.21.15.143:0/1165957718 wait complete. 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.617+0000 7ffb69d3d640 10 monclient: build_initial_monmap 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.617+0000 7ffb69d3d640 10 monclient: monmap: 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:epoch 0 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:last_changed 0.000000 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:created 0.000000 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:24.619 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 1 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.noname-a 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.noname-c 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.617+0000 7ffb69d3d640 1 librados: starting msgr at 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.617+0000 7ffb69d3d640 1 librados: starting objecter 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.617+0000 7ffb69d3d640 1 Processor -- start 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 1 -- start start 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 1 librados: setting wanted keys 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 1 librados: calling monclient init 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 10 monclient: init 2024-03-25T19:41:24.620 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb640b1730 addr v1:172.21.15.143:6789/0 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb640b4d60 addr v1:172.21.15.203:6789/0 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb6414be90 addr v1:172.21.15.143:6790/0 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 1 -- --> v1:172.21.15.143:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb641419c0 con 0x7ffb640b1730 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6413bbc0 con 0x7ffb640b4d60 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 1 -- --> v1:172.21.15.143:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6413bd50 con 0x7ffb6414be90 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:24.621 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.618+0000 7ffb69d3d640 10 monclient(hunting): authenticate will time out at 6700.962457s 2024-03-25T19:41:24.630 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.631+0000 7ffb63fff640 1 --1- >> v1:172.21.15.143:6790/0 conn(0x7ffb6414be90 0x7ffb641e20d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.143:6790/0 says I am v1:172.21.15.143:43422/0 (socket says 172.21.15.143:43422) 2024-03-25T19:41:24.630 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.631+0000 7ffb63fff640 1 -- 172.21.15.143:0/3525188390 learned_addr learned my addr 172.21.15.143:0/3525188390 (peer_addr_for_me v1:172.21.15.143:0/0) 2024-03-25T19:41:24.631 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.632+0000 7ffb4bfff640 10 client.?.objecter ms_handle_connect 0x7ffb6414be90 2024-03-25T19:41:24.631 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.632+0000 7ffb4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:24.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.633+0000 7ffb4bfff640 10 client.?.objecter ms_handle_connect 0x7ffb640b1730 2024-03-25T19:41:24.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.633+0000 7ffb4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:24.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.633+0000 7ffb4bfff640 10 client.?.objecter ms_handle_connect 0x7ffb640b4d60 2024-03-25T19:41:24.633 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.633+0000 7ffb4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:24.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.634+0000 7ffb637fe640 1 -- 172.21.15.143:0/3525188390 >> v1:172.21.15.143:6789/0 conn(0x7ffb640b1730 legacy=0x7ffb641405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:41:24.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.634+0000 7ffb637fe640 1 -- 172.21.15.143:0/3525188390 >> v1:172.21.15.143:6789/0 conn(0x7ffb640b1730 legacy=0x7ffb641405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:41:24.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.634+0000 7ffb637fe640 1 --1- 172.21.15.143:0/3525188390 >> v1:172.21.15.143:6789/0 conn(0x7ffb640b1730 0x7ffb641405f0 :-1 s=OPENED pgs=87410 cs=1 l=1).handle_message read tag failed 2024-03-25T19:41:24.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.634+0000 7ffb637fe640 1 --1- 172.21.15.143:0/3525188390 >> v1:172.21.15.143:6789/0 conn(0x7ffb640b1730 0x7ffb641405f0 :-1 s=OPENED pgs=87410 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:41:24.634 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.634+0000 7ffb4bfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.143:6789/0 2024-03-25T19:41:24.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.644+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577167224 0 0) 0x7ffb6413bd50 con 0x7ffb6414be90 2024-03-25T19:41:24.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.644+0000 7ffb4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.644+0000 7ffb4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.644+0000 7ffb4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.644+0000 7ffb4bfff640 10 monclient(hunting): my global_id is 4733 2024-03-25T19:41:24.644 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.644+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 --> v1:172.21.15.143:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50004eb0 con 0x7ffb6414be90 2024-03-25T19:41:24.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.649+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092775403 0 0) 0x7ffb6413bbc0 con 0x7ffb640b4d60 2024-03-25T19:41:24.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.649+0000 7ffb4bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:24.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.649+0000 7ffb4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:24.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.649+0000 7ffb4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:24.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.649+0000 7ffb4bfff640 10 monclient(hunting): my global_id is 4747 2024-03-25T19:41:24.649 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.649+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6413bd50 con 0x7ffb640b4d60 2024-03-25T19:41:24.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.666+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2408758819 0 0) 0x7ffb50004eb0 con 0x7ffb6414be90 2024-03-25T19:41:24.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.666+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 --> v1:172.21.15.143:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb6413bbc0 con 0x7ffb6414be90 2024-03-25T19:41:24.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.666+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ffb5c001f60 con 0x7ffb6414be90 2024-03-25T19:41:24.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.666+0000 7ffb4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.666+0000 7ffb4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:24.666 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.666+0000 7ffb4bfff640 10 monclient(hunting): dump: 2024-03-25T19:41:24.666 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:24.666 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:24.667 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.666+0000 7ffb4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:24.677 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1459536986 0 0) 0x7ffb6413bbc0 con 0x7ffb6414be90 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 >> v1:172.21.15.203:6789/0 conn(0x7ffb640b4d60 legacy=0x7ffb641d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 >> v1:172.21.15.143:6789/0 conn(0x7ffb640b1730 legacy=0x7ffb641405f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 1 monclient: found mon.c 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 --> v1:172.21.15.143:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6413b060 con 0x7ffb6414be90 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.677+0000 7ffb4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 5 monclient: authenticate success, global_id 4733 2024-03-25T19:41:24.678 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 10 monclient: _renew_subs 2024-03-25T19:41:24.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:24.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 --> v1:172.21.15.143:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb641421a0 con 0x7ffb6414be90 2024-03-25T19:41:24.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:24.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 10 monclient: _renew_subs 2024-03-25T19:41:24.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:24.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 --> v1:172.21.15.143:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb641d8590 con 0x7ffb6414be90 2024-03-25T19:41:24.679 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.678+0000 7ffb69d3d640 1 librados: init done 2024-03-25T19:41:24.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.679+0000 7ffb69d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:24.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.679+0000 7ffb69d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:24.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.679+0000 7ffb69d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:24.680 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.679+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6413b250 con 0x7ffb6414be90 2024-03-25T19:41:24.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.995+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb5c003c20 con 0x7ffb6414be90 2024-03-25T19:41:24.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.995+0000 7ffb4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:24.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.995+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1941361977 0 0) 0x7ffb5c003fb0 con 0x7ffb6414be90 2024-03-25T19:41:24.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.995+0000 7ffb4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:24.995 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.995+0000 7ffb4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.995+0000 7ffb4bfff640 10 monclient: dump: 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:epoch 1 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:fsid d1ef59af-7ebc-4de5-a205-9cc943985e2f 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:last_changed 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:created 2024-03-25T19:29:00.532941+0000 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:election_strategy: 3 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:0: v1:172.21.15.143:6789/0 mon.a 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:2: v1:172.21.15.143:6790/0 mon.c 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr: 2024-03-25T19:41:24.996 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.995+0000 7ffb61ffb640 4 set_mon_vals no callback set 2024-03-25T19:41:24.997 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.996+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4217197518 0 0) 0x7ffb5c0041d0 con 0x7ffb6414be90 2024-03-25T19:41:24.997 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.996+0000 7ffb637fe640 10 monclient: get_auth_request con 0x7ffb5006dd00 auth_method 0 2024-03-25T19:41:24.997 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:24.997+0000 7ffb4bfff640 10 client.4733.objecter ms_handle_connect 0x7ffb5006dd00 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.002+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1273868597 0 0) 0x7ffb5c054a90 con 0x7ffb6414be90 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.002+0000 7ffb4bfff640 10 client.4733.objecter ms_dispatch 0x7ffb64000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.002+0000 7ffb4bfff640 3 client.4733.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.002+0000 7ffb4bfff640 3 client.4733.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.002+0000 7ffb4bfff640 20 client.4733.objecter dump_active .. 0 homeless 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.002+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb5c0505a0 con 0x7ffb6414be90 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.002+0000 7ffb4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:25.002 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.002+0000 7ffb4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:25.228 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.229+0000 7ffb69d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.229+0000 7ffb69d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.229+0000 7ffb69d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.143:6790/0 2024-03-25T19:41:25.229 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.229+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 --> v1:172.21.15.143:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb640627b0 con 0x7ffb6414be90 2024-03-25T19:41:25.230 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:51.795 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.239 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:51.804 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.280 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:51.845 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.281 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:11:51.846 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.295 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:51.831 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.301+0000 7ffb4bfff640 1 -- 172.21.15.143:0/3525188390 <== mon.2 v1:172.21.15.143:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3421209569) 0x7ffb5c0524d0 con 0x7ffb6414be90 2024-03-25T19:41:25.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.301+0000 7ffb4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:25.301 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.301+0000 7ffb4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:25.301 INFO:teuthology.misc.health.smithi143.stdout: 2024-03-25T19:41:25.302 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":11,"fsid":"d1ef59af-7ebc-4de5-a205-9cc943985e2f","created":"2024-03-25T19:29:55.704800+0000","modified":"2024-03-25T19:39:59.588612+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:39:59.588612+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"822e7ad9-f685-4d65-981e-8366577fa717","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"388a4708-38b7-4278-a660-a79b90807cbe","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"5b4e989e-4066-411c-8d76-23d9e8f27d83","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"0490eb42-6979-4362-9b8c-efaeb3e7bb54","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf6a03da-5706-4ec1-aca2-b21e11e3c8ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"44b74cce-f2bf-4da3-bf62-8768784cad61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a5ddf86-73e1-4c36-96da-8b4707ebb1c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"506e8acc-eec5-4d6c-b8af-448710399c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:25.302 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:51.838 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.304+0000 7ffb69d3d640 10 librados: watch_flush enter 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.304+0000 7ffb69d3d640 10 librados: watch_flush exit 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.304+0000 7ffb69d3d640 20 client.4733.objecter shutdown clearing up homeless session... 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.304+0000 7ffb69d3d640 10 client.4733.objecter successfully canceled tick 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 >> v1:172.21.15.143:6800/1292282666 conn(0x7ffb5006dd00 legacy=0x7ffb500701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 10 monclient: shutdown 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 >> v1:172.21.15.143:6790/0 conn(0x7ffb6414be90 legacy=0x7ffb641e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 shutdown_connections 2024-03-25T19:41:25.305 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 >> 172.21.15.143:0/3525188390 conn(0x7ffb640617a0 msgr2=0x7ffb6413c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:25.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 shutdown_connections 2024-03-25T19:41:25.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 1 -- 172.21.15.143:0/3525188390 wait complete. 2024-03-25T19:41:25.306 INFO:teuthology.misc.health.smithi143.stderr:2024-03-25T19:41:25.305+0000 7ffb69d3d640 1 librados: shutdown 2024-03-25T19:41:25.308 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:51.873 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:25.318 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:41:25.319 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:41:25.320 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:41:25.320 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:25.345 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:11:51.910 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:25.352 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:11:51.888 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:27.258 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:11:25.436 35814== 2024-03-25T19:41:27.258 INFO:tasks.ceph.osd.0.smithi143.stderr:==00:00:11:25.436 35814== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:27.289 INFO:tasks.ceph.osd.0.smithi143.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:31.421 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi143 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:41:31.422 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:41:31.422 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:41:31.422 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:33.289 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:11:31.443 35818== 2024-03-25T19:41:33.289 INFO:tasks.ceph.osd.1.smithi143.stderr:==00:00:11:31.443 35818== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:33.328 INFO:tasks.ceph.osd.1.smithi143.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:37.524 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi143 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:41:37.524 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:41:37.525 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:41:37.525 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:39.495 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:11:37.649 35817== 2024-03-25T19:41:39.495 INFO:tasks.ceph.osd.2.smithi143.stderr:==00:00:11:37.649 35817== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:39.531 INFO:tasks.ceph.osd.2.smithi143.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:43.627 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi143 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:41:43.627 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:41:43.628 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:41:43.628 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:45.030 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:12:11.595 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.040 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:12:11.605 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.068 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:12:11.633 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.084 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:11.619 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.086 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:11.622 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.090 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:12:11.655 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.097 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:12:11.662 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.113 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:12:11.678 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.115 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:11.651 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:45.391 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:11:43.547 35816== 2024-03-25T19:41:45.391 INFO:tasks.ceph.osd.3.smithi143.stderr:==00:00:11:43.547 35816== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:45.437 INFO:tasks.ceph.osd.3.smithi143.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:49.729 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi143 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:41:49.730 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:41:49.730 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:41:49.730 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:51.821 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:11:49.995 35943== 2024-03-25T19:41:51.821 INFO:tasks.ceph.osd.4.smithi203.stderr:==00:00:11:49.995 35943== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:51.857 INFO:tasks.ceph.osd.4.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:41:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:41:55.832 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:41:55.832 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:41:55.833 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:41:55.833 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:41:57.794 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:11:55.970 35944== 2024-03-25T19:41:57.794 INFO:tasks.ceph.osd.5.smithi203.stderr:==00:00:11:55.970 35944== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:41:57.838 INFO:tasks.ceph.osd.5.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:42:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:42:01.935 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:42:01.935 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:42:01.935 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:42:01.935 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:42:03.819 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:12:01.993 35945== 2024-03-25T19:42:03.819 INFO:tasks.ceph.osd.6.smithi203.stderr:==00:00:12:01.993 35945== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:42:03.861 INFO:tasks.ceph.osd.6.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:42:05.066 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:12:31.631 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.067 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:12:31.632 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.082 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:12:31.648 35441== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.113 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:31.648 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.113 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:12:31.678 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.115 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:12:31.680 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.115 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:31.651 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.128 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:12:31.693 35443== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:05.128 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:12:31.664 35398== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:42:08.038 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:42:08.039 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:42:08.039 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:42:08.039 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:42:09.582 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:12:07.738 35946== 2024-03-25T19:42:09.582 INFO:tasks.ceph.osd.7.smithi203.stderr:==00:00:12:07.738 35946== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:42:09.612 INFO:tasks.ceph.osd.7.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:42:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:42:14.141 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:42:14.141 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:42:14.141 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:42:14.142 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:42:14.142 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:42:14.169 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:42:14.169 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:42:14.169 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:42:14.196 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:42:14.196 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:42:14.196 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:42:14.196 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:42:14.224 INFO:tasks.ceph.mon.a.smithi143.stderr:2024-03-25T19:42:14.213+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35408) UID: 0 2024-03-25T19:42:14.225 INFO:tasks.ceph.mon.a.smithi143.stderr:2024-03-25T19:42:14.215+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:44:42.847 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:15:09.412 35441== 2024-03-25T19:44:42.848 INFO:tasks.ceph.mon.a.smithi143.stderr:==00:00:15:09.412 35441== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:44:42.856 INFO:tasks.ceph.mon.a.smithi143.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:44:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:44:46.732 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi143 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:44:46.733 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:44:46.733 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:44:46.733 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:44:46.751 INFO:tasks.ceph.mon.c.smithi143.stderr:2024-03-25T19:44:46.738+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35410) UID: 0 2024-03-25T19:44:46.751 INFO:tasks.ceph.mon.c.smithi143.stderr:2024-03-25T19:44:46.740+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:44:47.232 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:15:13.797 35443== 2024-03-25T19:44:47.232 INFO:tasks.ceph.mon.c.smithi143.stderr:==00:00:15:13.797 35443== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:44:47.241 INFO:tasks.ceph.mon.c.smithi143.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:44:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:44:52.835 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi143 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:44:52.835 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:44:52.836 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:44:52.836 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:44:52.856 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-03-25T19:44:52.841+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35394) UID: 0 2024-03-25T19:44:52.856 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-03-25T19:44:52.842+0000 9f86640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:44:53.484 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:20.020 35398== 2024-03-25T19:44:53.484 INFO:tasks.ceph.mon.b.smithi203.stderr:==00:00:15:20.020 35398== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:44:53.496 INFO:tasks.ceph.mon.b.smithi203.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:44:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:44:58.938 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:44:58.938 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:44:58.939 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:44:58.939 DEBUG:teuthology.orchestra.run.smithi143:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:44:58.983 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:44:58.983 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:44:59.079 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:44:59.079 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:44:59.175 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:44:59.175 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:44:59.235 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:44:59.235 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:44:59.310 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:44:59.310 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:44:59.425 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:44:59.425 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:44:59.487 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:44:59.487 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:44:59.545 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:44:59.545 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:44:59.604 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:44:59.606 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/data/mon.a.tgz 2024-03-25T19:44:59.606 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-03-25T19:44:59.623 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.PqECJQNUeN 2024-03-25T19:44:59.624 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.PqECJQNUeN 2024-03-25T19:44:59.724 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.PqECJQNUeN 2024-03-25T19:44:59.810 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.PqECJQNUeN is 133KB 2024-03-25T19:44:59.862 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.PqECJQNUeN 2024-03-25T19:44:59.876 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/data/mon.c.tgz 2024-03-25T19:44:59.877 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-03-25T19:44:59.931 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.MHpqg3mh8l 2024-03-25T19:44:59.931 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.MHpqg3mh8l 2024-03-25T19:45:00.024 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.MHpqg3mh8l 2024-03-25T19:45:00.108 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.MHpqg3mh8l is 164KB 2024-03-25T19:45:00.160 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.MHpqg3mh8l 2024-03-25T19:45:00.174 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/data/mon.b.tgz 2024-03-25T19:45:00.175 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-25T19:45:00.192 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.MMRHBdf8NF 2024-03-25T19:45:00.193 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.MMRHBdf8NF 2024-03-25T19:45:00.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.MMRHBdf8NF 2024-03-25T19:45:00.388 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.MMRHBdf8NF is 155KB 2024-03-25T19:45:00.440 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.MMRHBdf8NF 2024-03-25T19:45:00.455 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:45:00.455 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:45:00.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:45:00.783 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:45:01.053 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:45:01.055 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/remote/smithi143/crash 2024-03-25T19:45:01.056 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:45:01.082 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/remote/smithi203/crash 2024-03-25T19:45:01.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:45:01.110 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:45:01.110 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:45:01.126 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41351.log 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39947.log 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41351.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39947.log: gzip 86.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.41351.log.gz 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40259.log 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39752.log 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39947.log.gz 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41117.log 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42601.log 2024-03-25T19:45:01.196 INFO:teuthology.orchestra.run.smithi143.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40259.log.gz 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41975.log 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39752.log.gz 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40181.log 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41117.log.gz 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42601.log.gz 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41819.log 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39360.log 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41975.log.gz 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40727.log 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40181.log.gz 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39360.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41819.log.gz 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39399.log 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40727.log: /var/log/ceph/ceph-client.admin.40688.log 2024-03-25T19:45:01.197 INFO:teuthology.orchestra.run.smithi143.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39360.log.gz 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40922.log 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40727.log.gz 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40688.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39399.log.gz 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5/var/log/ceph/ceph-client.admin.40922.log: --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40688.log.gz 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40922.log.gz 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/valgrind/osd.0.log: /var/log/ceph/valgrind/osd.1.log: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:45:01.198 INFO:teuthology.orchestra.run.smithi143.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39869.log 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40766.log 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39557.log 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39869.log.gz 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40103.log 86.7% 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40766.log.gz 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41312.log 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39557.log.gz 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35450.log 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41312.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40103.log.gz 2024-03-25T19:45:01.199 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40961.log 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.35450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42406.log 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41312.log.gz 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41156.log 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42406.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35450.log.gz 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40844.log 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40961.log.gz 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41156.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42406.log.gz 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39440.log 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40220.log 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41156.log.gz 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40844.log.gz 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40220.log: gzip -5 --verbose 86.8% -- -- replaced with /var/log/ceph/ceph-client.admin.39440.log.gz /var/log/ceph/ceph-client.admin.42131.log 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:45:01.200 INFO:teuthology.orchestra.run.smithi143.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40220.log.gz 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41897.log 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35860.log 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gz 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41897.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42131.log.gz 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41273.log 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.35860.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41897.log.gz 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41429.log 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40454.log 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35860.log.gz 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41780.log 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41273.log.gz 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40454.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41429.log.gz 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39518.log 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40376.log 2024-03-25T19:45:01.201 INFO:teuthology.orchestra.run.smithi143.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40454.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39830.log 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41780.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40883.log 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39518.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39830.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40376.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40883.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39830.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42092.log 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40883.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42523.log 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42367.log 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42523.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42092.log.gz 2024-03-25T19:45:01.202 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41663.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42367.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42523.log.gz 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41936.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40415.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42367.log.gz 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42484.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41663.log.gz 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40415.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41936.log.gz 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36011.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42484.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40415.log.gz 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42210.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.36011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42289.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42484.log.gz 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36011.log.gz 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36871.log 2024-03-25T19:45:01.203 INFO:teuthology.orchestra.run.smithi143.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42210.log.gz 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42289.log.gz 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.36871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42014.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36871.log.gz/var/log/ceph/ceph-client.admin.41507.log: 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37843.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35813.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42014.log.gz 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.37843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.35813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41468.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37843.log.gz 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35813.log.gz 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5/var/log/ceph/ceph-client.admin.40025.log: --verbose -- /var/log/ceph/ceph-client.admin.41624.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42328.log 2024-03-25T19:45:01.204 INFO:teuthology.orchestra.run.smithi143.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39635.log 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41468.log.gz 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42171.log 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41624.log.gz 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40142.log 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42328.log.gz 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39635.log.gz 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42171.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40337.log 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-osd.3.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42171.log.gz 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.40142.log.gz -5 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42250.log 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40337.log: /var/log/ceph/ceph-client.admin.42250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42445.log 2024-03-25T19:45:01.205 INFO:teuthology.orchestra.run.smithi143.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40337.log.gz 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41195.log: gzip -5 --verbose 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42250.log.gz 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr: -- /var/log/ceph/ceph-client.admin.39713.log 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42445.log: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz -5 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40298.log 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39713.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42445.log.gz 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39596.log 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41000.log 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39713.log.gz 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40649.log 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40298.log.gz 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42562.log 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39596.log.gz 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40649.log: gzip -5 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41000.log.gz 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2024-03-25T19:45:01.206 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42562.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40649.log.gz 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39139.log: gzip -5 --verbose -- 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42562.log.gz 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr: /var/log/ceph/ceph-client.admin.41234.log 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-mgr.y.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29584.log 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41234.log: /var/log/ceph/ceph.tmp-client.admin.29584.log: gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29584.log.gz 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40610.log 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41234.log.gz 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40805.log 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39674.log 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.40805.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40610.log.gz 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:45:01.207 INFO:teuthology.orchestra.run.smithi143.stderr:gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40805.log.gz 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39674.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40064.log 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41585.log 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr: 88.6%/var/log/ceph/ceph-client.admin.40064.log: -- replaced with /var/log/ceph/ceph-client.admin.39674.log.gz 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40064.log.gz 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41585.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41585.log.gz/var/log/ceph/ceph-osd.2.log: 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41546.log 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41078.log: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39791.log 2024-03-25T19:45:01.208 INFO:teuthology.orchestra.run.smithi143.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41546.log.gz 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42640.log 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.39791.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39791.log.gz 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41858.log 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.42640.log: /var/log/ceph/ceph-client.admin.40532.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42640.log.gz 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41039.log 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42053.log 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41858.log: /var/log/ceph/ceph-client.admin.41039.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41858.log.gz 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41039.log.gz 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41741.log 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.42053.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42053.log.gz 2024-03-25T19:45:01.209 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph-client.admin.41741.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41741.log.gz 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35591.log 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr:gzip/var/log/ceph/ceph-client.admin.35591.log: -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35591.log.gz 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.5.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:45:01.211 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35711.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35711.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35711.log.gz 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35551.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35631.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.35551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35671.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35631.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.35551.log.gz 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:45:01.212 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35751.log 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35631.log.gz 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35671.log.gz 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35792.log 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35751.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.35792.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35751.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35844.log 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35792.log.gz 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.35844.log: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35844.log.gz 2024-03-25T19:45:01.213 INFO:teuthology.orchestra.run.smithi203.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:45:01.251 INFO:teuthology.orchestra.run.smithi143.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:45:02.037 INFO:teuthology.orchestra.run.smithi143.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:45:02.085 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:45:02.113 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:45:02.292 INFO:teuthology.orchestra.run.smithi143.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:45:02.310 INFO:teuthology.orchestra.run.smithi143.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:45:02.382 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:45:02.383 INFO:teuthology.orchestra.run.smithi143.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:45:02.415 INFO:teuthology.orchestra.run.smithi203.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:45:06.194 INFO:teuthology.orchestra.run.smithi203.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:45:06.196 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-25T19:45:06.196 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m5.026s 2024-03-25T19:45:06.196 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m8.959s 2024-03-25T19:45:06.196 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.342s 2024-03-25T19:45:06.253 INFO:teuthology.orchestra.run.smithi143.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:45:06.407 INFO:teuthology.orchestra.run.smithi143.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:45:06.409 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-25T19:45:06.409 INFO:teuthology.orchestra.run.smithi143.stderr:real 0m5.271s 2024-03-25T19:45:06.409 INFO:teuthology.orchestra.run.smithi143.stderr:user 0m14.301s 2024-03-25T19:45:06.409 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m0.530s 2024-03-25T19:45:06.409 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:45:06.409 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/remote/smithi143/log 2024-03-25T19:45:06.410 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:45:08.804 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/remote/smithi203/log 2024-03-25T19:45:08.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:45:10.291 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:45:10.496 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c88a0e4328724672814c7e86f70ec1b3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:45:10.500 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:45:10.509 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:45:10.518 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:45:10.518 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:45:10.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:45:10.612 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:45:10.622 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:45:10.622 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:45:10.624 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:45:10.639 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:45:10.641 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:45:10.644 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:45:10.644 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-03-25T19:45:10.644 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 -393us[ -331us] +/- 71ms 2024-03-25T19:45:10.644 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv02.front.sepia.ceph.com 3 6 377 41 -309us[ -248us] +/- 57ms 2024-03-25T19:45:10.644 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +748us[ +808us] +/- 61ms 2024-03-25T19:45:10.644 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:45:10.645 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:45:10.645 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-25T19:45:10.646 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 -403us[ -320us] +/- 71ms 2024-03-25T19:45:10.646 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -326us[ -243us] +/- 57ms 2024-03-25T19:45:10.646 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +757us[ +840us] +/- 61ms 2024-03-25T19:45:10.646 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:45:10.647 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:45:10.656 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:45:10.657 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:45:10.668 DEBUG:teuthology.orchestra.run.smithi143:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:45:10.755 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:45:10.826 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:45:10.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:45:10.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:45:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:45:10.889 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:45:10.897 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:45:10.908 INFO:teuthology.task.internal:Duration was 6603.135285 seconds 2024-03-25T19:45:10.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:45:10.941 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:45:10.942 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:45:10.944 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:45:10.986 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:45:10.989 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:45:11.320 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:45:11.321 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-03-25T19:45:11.321 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:45:11.381 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-03-25T19:45:11.381 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:45:11.410 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:45:11.410 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:45:11.424 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:45:11.479 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:45:11.480 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:45:11.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:45:11.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:45:11.634 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:45:11.635 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:45:11.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:45:11.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:45:11.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:45:11.705 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:45:11.727 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-03-25T19:45:11.755 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-03-25T19:45:11.807 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:45:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:45:11.855 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:45:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:45:11.869 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:45:11.879 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:45:11.880 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/remote/smithi143 2024-03-25T19:45:11.880 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:45:11.933 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309/remote/smithi203 2024-03-25T19:45:11.933 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:45:11.967 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:45:11.967 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:45:11.970 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:45:12.017 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:45:12.028 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:45:12.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:45:12.044 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:45:12.044 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:45:12.046 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:45:12.058 INFO:teuthology.orchestra.run.smithi143.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:45 /home/ubuntu/cephtest 2024-03-25T19:45:12.072 INFO:teuthology.orchestra.run.smithi203.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:45 /home/ubuntu/cephtest 2024-03-25T19:45:12.073 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:45:12.085 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:45:12.117 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:45:12.142 DEBUG:teuthology.nuke:shortname: smithi143 2024-03-25T19:45:12.143 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:45:12.164 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:11.561303', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUEFSADnVvILqzvtrWtHAF9jYxUwPIXtnJtROHs2BTLVMNONytR0y9dNqmYCAahSYtKDAkrpvk18RnY3FK4g8Q='} 2024-03-25T19:45:12.167 DEBUG:teuthology.nuke:shortname: smithi203 2024-03-25T19:45:12.167 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:45:12.190 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621309', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:46:11.562925', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBq7nuCIIYPLhS6F1kW7ValjeH/QJoJLAYq3KIIS6zBJhjXFs5rWzRdx5QRsovxS0X92BplHJpKsJ1Ln2Eo+s/M='} 2024-03-25T19:45:12.218 INFO:teuthology.orchestra.console.smithi143:Power off 2024-03-25T19:45:12.218 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:45:12.243 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off 2024-03-25T19:45:12.244 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:45:12.259 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-25T19:45:12.260 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:45:12.268 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-25T19:45:12.281 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-25T19:45:12.281 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:45:12.303 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-25T19:45:16.269 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:45:16.304 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:45:16.378 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-25T19:45:16.414 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-25T19:45:20.382 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:45:20.415 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:45:20.493 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-25T19:45:20.525 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-25T19:45:24.498 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:45:24.525 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:45:24.609 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off 2024-03-25T19:45:24.637 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-25T19:45:24.709 INFO:teuthology.orchestra.console.smithi143:Power off completed 2024-03-25T19:45:24.738 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-25T19:45:24.843 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6603.135284662247 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c88a0e4328724672814c7e86f70ec1b3 status: fail success: false 2024-03-25T19:45:24.843 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:45:24.921 INFO:teuthology.run:FAIL